Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka >> mail # user >> Consumers can't connect while broker is under load


+
Tim Kellogg 2014-01-11, 02:03
+
Guozhang Wang 2014-01-11, 02:21
Copy link to this message
-
Re: Consumers can't connect while broker is under load
12 zookeepers seems like a lot......and you should always, by default,
prefer an odd number of zookeepers.  Consumers negotiate with each
other for partition ownership, via zookeeper.....

Jason

On Fri, Jan 10, 2014 at 9:20 PM, Guozhang Wang <[EMAIL PROTECTED]> wrote:
> Can you post the consumer logs for the long-time-to-connect scenarios?
>
> Guozhang
>
>
> On Fri, Jan 10, 2014 at 1:20 PM, Tim Kellogg <[EMAIL PROTECTED]> wrote:
>
>> Hi,
>>
>> I have a cluster of 12 brokers receiving 10,000 msg/s from producers where
>> each message is roughly 2.5KB. We also have 12 ZooKeepers and everything is
>> on AWS. Under these conditions, top (the Linux utility) reports around
>> 10-15 out of 32 for system load, so we’re at less than half capacity.
>>
>> When under this load consumers take a very long time, often more than 30
>> minutes, to connect to the brokers. When under no load they connect
>> immediately. Why is this happening?
>>
>> Thanks,
>>
>> Tim Kellogg
>> Sr. Software Engineer, Protocols
>> 2lemetry
>> @kellogh
>
>
>
>
> --
> -- Guozhang

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB