Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka >> mail # dev >> Kafka stream processing framework?


+
David Arthur 2012-11-08, 15:32
+
Prashanth Menon 2012-11-08, 15:45
Copy link to this message
-
Re: Kafka stream processing framework?
Prashanth,

Storm seems to be more focused on data flow between Storm processors (spout? bolt? i forget). My particular use case follows this pattern:

* read id from kafka queue
* fetch object from database
* modify the object
* write back to database

Would Storm be a good fit for this? It doesn't seem to fit in with the whole bolt/spout pattern. It's more like a distributed task queue.

Thoughts?

On Nov 8, 2012, at 10:45 AM, Prashanth Menon wrote:

> Yup, I believe Storm as a KafkaSpout that you can use.  Is there something
> specific you were interested in?
>
> On Thu, Nov 8, 2012 at 10:32 AM, David Arthur <[EMAIL PROTECTED]> wrote:
>
>> There is a line item on the project ideas for "improved stream processing
>> libraries". I was wondering if anyone has done any work on this. I know you
>> can hook Kafka into things like Storm and S4(?), but I'm not looking for a
>> CEP/dataflow thing, just distributed stream processing
>>
>> -David
+
Prashanth Menon 2012-11-08, 20:37
+
Milind Parikh 2012-11-08, 22:56
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB