Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Documentation for metadata.broker.list


Copy link to this message
-
Documentation for metadata.broker.list
I recently moved my 0.8.0 cluster to a set of entirely new brokers, and was
surprised to find that the producers did not update their list of brokers
to remove the brokers that were no longer in the cluster. That is, I had
brokers 1,2,3,4,5, added brokers 6,7,8,910, waited a day, and stopped
brokers 1,2,3,4,5. After stopping the original brokers, the producers
continued trying to fetch metadata from them, even though they no longer
appeared the topic metadata. It looks like it was basically the same issue
described in
http://mail-archives.apache.org/mod_mbox/kafka-users/201402.mbox/%[EMAIL PROTECTED]%3E
.

The documentation for "metadata.broker.list" says "This is for
bootstrapping and the producer will only use it for getting metadata". I,
and apparently others, have interpreted the "this is for bootstrapping"
part to mean that it is only used for the initial metadata fetch, and not
as the sole list of brokers to use to fetch metadata.

Does it make sense to change the documentation to something like "The list
of brokers used to fetch metadata. These brokers must always be available"?

Am I understanding metadata.broker.list correctly? Is it necessary to load
balance these brokers or otherwise make sure it does not refer directly to
any brokers that could be removed?

Thanks,

Ryan

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB