Kafka, mail # user - Re: About kafka 0.8 producer auto detect broker - 2013-01-15, 17:59
Solr & Elasticsearch trainings in New York & San Francisco [more info][hide]
 Search Hadoop and all its subprojects:

Switch to Plain View
Jun Rao 2013-01-15, 15:24
gj1989lh 2013-01-15, 14:42
gj1989lh 2013-01-15, 07:02
Copy link to this message
Re: About kafka 0.8 producer auto detect broker

In 0.8, we got rid of zookeeper from the producer and replaced its
functionality with a metadata API. On startup and whenever a request fails,
the producer refreshes its view of the cluster by sending a metadata
request to any of the brokers. So if a broker goes down or a new broker
comes up, the leader for some partitions might change and the producer will
know since its requests to the older leaders will fail.

By removing the dependency of zookeeper from the producer, we have in fact
helped scale the number of producers you can have in a cluster since now
each of them don't need to have a connection to zookeeper and deal with
session expirations when there are minor glitches on the zookeeper cluster.
Another goal of this change was also to enable writing non-java clients


gj1989lh 2013-01-15, 07:02
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB