Kafka, mail # user - Re: Strategies for auto generating broker ID - 2013-10-02, 18:11
 Search Hadoop and all its subprojects:

Switch to Threaded View
Copy link to this message
-
Re: Strategies for auto generating broker ID
I recently moved away from generating a unique brokerId for each node, in
favor of assigning ids in configuration.  The reason for this, is that in
0.8, there isn't a convenient way yet to reassign partitions to a new
brokerid, should one broker have a failure.  So, it seems the only
work-around at the moment is to bring up a replacement broker, assign it
the same brokerId as one that has failed and is no longer running.  The
cluster will then automatically replicate all the partitions that were
assigned to the failed broker to the new broker.

This appears the only operational way to deal with failed brokers, at the
moment.

Longer term, it would be great if the cluster were self-healing, and if a
broker went down, we could mark it as no longer available somehow, and the
cluster would then reassign and re-replicate partitions to new brokers,
that were previously assigned to the failed broker.  I expect something
like this will be available in future versions, but that doesn't appear the
case at present.

And related, it would be nice, in the interests of horizontal scalability,
to have an easy way for the cluster to dynamically rebalance load, if new
nodes are added to the cluster (or to at least prefer assigning new
partitions to brokers which have more space available).  I expect this will
be something to prioritize in the future versions as well.

Jason
On Wed, Oct 2, 2013 at 1:00 PM, Sriram Subramanian <
[EMAIL PROTECTED]> wrote:
 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB