Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka >> mail # user >> random access performance of messages.


+
marko@... 2011-10-21, 16:02
Copy link to this message
-
Re: random access performance of messages.
kafka is more suited for sequential message reads. Not really meant for
random message lookups.

Also using kafka as *long* term message store is not a good usecase.

On Fri, Oct 21, 2011 at 9:32 PM, <[EMAIL PROTECTED]> wrote:

> I would like to use Kafka to process messages that need to be immutably
> stored for a N-days, and during that period the msgs need to be indexed,
> searched, as well as retrieval of msg data that is queried.
>
>
>
> One approach is to read messages from Kafka and store the messages in a
> secondary db for query and data retrieval.  Once the messages are read and
> processed into the secondary db, then the messages can be discarded from
> the
> Kafka queue.
>
>
>
> Another approach is to read the messages, build an external index for
> searching that directly references the message data by Kafka-key in the
> Kafka queue itself.  In this case the Kafka becomes the message store for
> the life of the message/data.
>
>
>
> The latter would be ideal for me if the performance of query-by-key and
> message data retrieval is very good.
>
>
>
> Is random query of message+data good for Kafka?  Is this an appropriate
> usecase for Kafka?
>
>
>
> Thank you.
>
>
>
> Marko.
>
> .
>
>
>
>
--
Thanks
Sharad Agarwal
Hadoop and Avro Committer
Technology Platforms, InMobi
*Disclaimer: Opinions expressed here are my own and do not represent past or
present employers.*
+
Neha Narkhede 2011-10-21, 17:02
+
marko@... 2011-10-21, 17:19
+
Neha Narkhede 2011-10-21, 22:35