Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka, mail # user - Thread Safety of KafkaStreams


Copy link to this message
-
Re: Thread Safety of KafkaStreams
Neha Narkhede 2012-11-21, 06:19
David,

One KafkaStream is meant to be iterated by a single thread. A better
approach is to request higher number of streams
from the Kafka consumer and let each process have its own KafkaStream.

Thanks,
Neha

On Tue, Nov 20, 2012 at 9:40 PM, David Ross <[EMAIL PROTECTED]> wrote:
> Hello,
>
> We want to process messages from a single KafkaStream in a number of
> processes. Is it possible to have this code executing in multiple threads
> against the same stream?
>
> for (message <- stream) {
>   someBlockingOperation(message)
> }
>
> The scaladocs mention thread safety, but some of the code seems fairly
> stateful. I was wondering if anyone has experience with this or knows if it
> will work?
>
>
> Thanks,
>
> David