Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> random access performance of messages.


Copy link to this message
-
Re: random access performance of messages.
kafka is more suited for sequential message reads. Not really meant for
random message lookups.

Also using kafka as *long* term message store is not a good usecase.

On Fri, Oct 21, 2011 at 9:32 PM, <[EMAIL PROTECTED]> wrote:

> I would like to use Kafka to process messages that need to be immutably
> stored for a N-days, and during that period the msgs need to be indexed,
> searched, as well as retrieval of msg data that is queried.
>
>
>
> One approach is to read messages from Kafka and store the messages in a
> secondary db for query and data retrieval.  Once the messages are read and
> processed into the secondary db, then the messages can be discarded from
> the
> Kafka queue.
>
>
>
> Another approach is to read the messages, build an external index for
> searching that directly references the message data by Kafka-key in the
> Kafka queue itself.  In this case the Kafka becomes the message store for
> the life of the message/data.
>
>
>
> The latter would be ideal for me if the performance of query-by-key and
> message data retrieval is very good.
>
>
>
> Is random query of message+data good for Kafka?  Is this an appropriate
> usecase for Kafka?
>
>
>
> Thank you.
>
>
>
> Marko.
>
> .
>
>
>
>
--
Thanks
Sharad Agarwal
Hadoop and Avro Committer
Technology Platforms, InMobi
*Disclaimer: Opinions expressed here are my own and do not represent past or
present employers.*
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB