Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Consumers can't connect while broker is under load


Copy link to this message
-
Re: Consumers can't connect while broker is under load
12 zookeepers seems like a lot......and you should always, by default,
prefer an odd number of zookeepers.  Consumers negotiate with each
other for partition ownership, via zookeeper.....

Jason

On Fri, Jan 10, 2014 at 9:20 PM, Guozhang Wang <[EMAIL PROTECTED]> wrote:
> Can you post the consumer logs for the long-time-to-connect scenarios?
>
> Guozhang
>
>
> On Fri, Jan 10, 2014 at 1:20 PM, Tim Kellogg <[EMAIL PROTECTED]> wrote:
>
>> Hi,
>>
>> I have a cluster of 12 brokers receiving 10,000 msg/s from producers where
>> each message is roughly 2.5KB. We also have 12 ZooKeepers and everything is
>> on AWS. Under these conditions, top (the Linux utility) reports around
>> 10-15 out of 32 for system load, so we’re at less than half capacity.
>>
>> When under this load consumers take a very long time, often more than 30
>> minutes, to connect to the brokers. When under no load they connect
>> immediately. Why is this happening?
>>
>> Thanks,
>>
>> Tim Kellogg
>> Sr. Software Engineer, Protocols
>> 2lemetry
>> @kellogh
>
>
>
>
> --
> -- Guozhang