Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka, mail # dev - Kafka stream processing framework?


+
David Arthur 2012-11-08, 15:32
+
Prashanth Menon 2012-11-08, 15:45
Copy link to this message
-
Re: Kafka stream processing framework?
David Arthur 2012-11-08, 16:38
Prashanth,

Storm seems to be more focused on data flow between Storm processors (spout? bolt? i forget). My particular use case follows this pattern:

* read id from kafka queue
* fetch object from database
* modify the object
* write back to database

Would Storm be a good fit for this? It doesn't seem to fit in with the whole bolt/spout pattern. It's more like a distributed task queue.

Thoughts?

On Nov 8, 2012, at 10:45 AM, Prashanth Menon wrote:

> Yup, I believe Storm as a KafkaSpout that you can use.  Is there something
> specific you were interested in?
>
> On Thu, Nov 8, 2012 at 10:32 AM, David Arthur <[EMAIL PROTECTED]> wrote:
>
>> There is a line item on the project ideas for "improved stream processing
>> libraries". I was wondering if anyone has done any work on this. I know you
>> can hook Kafka into things like Storm and S4(?), but I'm not looking for a
>> CEP/dataflow thing, just distributed stream processing
>>
>> -David
+
Prashanth Menon 2012-11-08, 20:37
+
Milind Parikh 2012-11-08, 22:56