Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka, mail # user - One 0.72 ConsumerConnector, multiple threads, 1 blocks. What happens?


+
Philip OToole 2013-06-13, 02:44
+
Jun Rao 2013-06-13, 04:09
+
Philip OToole 2013-06-13, 04:10
Copy link to this message
-
Re: One 0.72 ConsumerConnector, multiple threads, 1 blocks. What happens?
Philip O'Toole 2013-06-13, 04:19
Also, what is it in the ConsumerConnection that causes this behaviour?

I'm proposing here that we move to a model of ConsumerConnection per
thread. This will decouple the flow for each partition, and allow each
to flow, right? We only have one topic on the cluster.

Thanks,

Philip

On Wed, Jun 12, 2013 at 9:10 PM, Philip O'Toole <[EMAIL PROTECTED]> wrote:
> Jun -- thanks.
>
> But if the topic is the same, doesn't each thread get a partition?
> Isn't that how it works?
>
> Philip
>
> On Wed, Jun 12, 2013 at 9:08 PM, Jun Rao <[EMAIL PROTECTED]> wrote:
>> Yes, when the consumer is consuming multiple topics, if one thread stops
>> consuming topic 1, it can prevent new data getting into the consumer for
>> topic 2.
>>
>> Thanks,
>>
>> Jun
>>
>>
>> On Wed, Jun 12, 2013 at 7:43 PM, Philip O'Toole <[EMAIL PROTECTED]> wrote:
>>
>>> Hello -- we're using 0.72. We're looking at the source, but want to be
>>> sure. :-)
>>>
>>> We create a single ConsumerConnector, call createMessageStreams, and
>>> hand the streams off to individual threads. If one of those threads
>>> calls next() on a stream, gets some messages, and then *blocks* in
>>> some subsequent operation (and blocks for minutes), can it potentially
>>> cause all other threads (calling next() on other streams) to block
>>> too? Does something inside the ConsumerConnector block all other
>>> stream processing? This would explain some behaviour we're seeing.
>>>
>>> Thanks,
>>>
>>> Philip
>>>

 
+
Jun Rao 2013-06-13, 04:51
+
Philip OToole 2013-06-13, 12:36