Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka >> mail # user >> Thread Safety of KafkaStreams


+
David Ross 2012-11-21, 05:40
+
Neha Narkhede 2012-11-21, 06:19
+
David Arthur 2012-11-21, 13:55
Copy link to this message
-
Re: Thread Safety of KafkaStreams
Yes, this was a follow up question. I was wondering if I could cheaply
implement this by sharing the stream. Thanks for responses.

On Wednesday, November 21, 2012, David Arthur wrote:

> But you can only have as many KafkaStreams as there are partitions,
> correct?
>
> This was actually discussed on IRC yesterday. One solution mentioned
> was to consume messages with one consumer thread and fill an internal
> queue (e.g. java.util.concurrent.BlockingQueue) for N worker threads
> to read from.
>
> Sent from my phone
>
> On Nov 21, 2012, at 1:20 AM, Neha Narkhede <[EMAIL PROTECTED]<javascript:;>>
> wrote:
>
> > David,
> >
> > One KafkaStream is meant to be iterated by a single thread. A better
> > approach is to request higher number of streams
> > from the Kafka consumer and let each process have its own KafkaStream.
> >
> > Thanks,
> > Neha
> >
> > On Tue, Nov 20, 2012 at 9:40 PM, David Ross <[EMAIL PROTECTED]<javascript:;>>
> wrote:
> >> Hello,
> >>
> >> We want to process messages from a single KafkaStream in a number of
> >> processes. Is it possible to have this code executing in multiple
> threads
> >> against the same stream?
> >>
> >> for (message <- stream) {
> >>  someBlockingOperation(message)
> >> }
> >>
> >> The scaladocs mention thread safety, but some of the code seems fairly
> >> stateful. I was wondering if anyone has experience with this or knows
> if it
> >> will work?
> >>
> >>
> >> Thanks,
> >>
> >> David
>
+
Jay Kreps 2012-11-21, 15:56
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB