I have a cluster of 12 brokers receiving 10,000 msg/s from producers where each message is roughly 2.5KB. We also have 12 ZooKeepers and everything is on AWS. Under these conditions, top (the Linux utility) reports around 10-15 out of 32 for system load, so we’re at less than half capacity.
When under this load consumers take a very long time, often more than 30 minutes, to connect to the brokers. When under no load they connect immediately. Why is this happening?
Tim Kellogg Sr. Software Engineer, Protocols 2lemetry @kellogh
12 zookeepers seems like a lot......and you should always, by default, prefer an odd number of zookeepers. Consumers negotiate with each other for partition ownership, via zookeeper.....
On Fri, Jan 10, 2014 at 9:20 PM, Guozhang Wang <[EMAIL PROTECTED]> wrote:
NEW: Monitor These Apps!
Apache Lucene, Apache Solr and all other Apache Software Foundation project and their respective logos are trademarks of the Apache Software Foundation.
Elasticsearch, Kibana, Logstash, and Beats are trademarks of Elasticsearch BV, registered in the U.S. and in other countries. This site and Sematext Group is in no way affiliated with Elasticsearch BV.
Service operated by Sematext