Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> About kafka 0.8 producer auto detect broker


Copy link to this message
-
About kafka 0.8 producer auto detect broker
Hi,
We know in Kafka 0.7, we can specify zk.connect. And with zookeeper, the producer can dynamically detect broker. But in Kafka 0.8, we can't specify zk.connect for producer. How does the producer in Kafka 0.8 auto detect broker?
I have done two experiments.
In first one, I configure the broker.list as broker1 and broker2, and set num.partitions=4 for all brokers. Then, I start broker 1 and broker 3. Without create topic1, I produce message directly, there are 4 partitions generated. Two in broker 1 and two in broker 3. How can the producer detect broker 3?
In second one, I configure the broker.list as broker 1 and broker2, and set num.partitions=4 for all brokers. Then I start broker 3 only. Without create topic2, I produce message directly. As a result ,there is no partition created for topic 2, and the producer report ConnectException, it says fetching topic metadata for topics [Set(topic 2)] from broker 3 failed (kafka.client.ClientUtils$)
Can I guess that the auto detecting ability for producer is weakened in Kafka 0.8? Why you get rid of zookeeper from producer in Kafka 0.8?
Many thanks and best regards!

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB