Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Question about auto-rebalancing


Copy link to this message
-
Re: Question about auto-rebalancing
Hello Siyuan,

For the automatic leader re-election, yes we are considering to make it
work. Could you file a JIRA for this issue?

For the high-level consumer's rebalancing logic, you can find it at

https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-CanIpredicttheresultsoftheconsumerrebabalance%3F

Guozhang
On Fri, Oct 11, 2013 at 11:06 AM, [EMAIL PROTECTED] <[EMAIL PROTECTED]> wrote:

> Hi Jun,
>
> Thanks for your reply, but in a real cluster, one broker could serve
> different topics and different partitions, the simple consumer only has
> knowledge of brokers that are available but it has no knowledge to decide
> which broker is best to pick up to consume messages.  If you don't choose
> carefully, multiple simple consumer might end up with reading from same
> node which is definitely not good for performance.
> Interesting thing is I find out there is
> command kafka-preferred-replica-election.sh which will try to equally
> distribute the leadership among different brokers, this is good that I can
> always let my simple consumer reads from leader broker(even it fails, the
> replica will pick up as leader which is fine).  But why don't kafka cluster
> run this command automatically when there is a broker change(up/down) in
> the cluster so that the leadership can always be equally distributed among
> different brokers ASAP?  I think it's very good for simple consumer to
> decide which broker is good to read from.
>
> Another question is I'm also curious how high-level consumer is balanced. I
> assume each high-level consumer know other consumers(int the same group)
> which broker they read message from and it can try to avoid those brokers
> and to pick up a free one?  Is there a document for the balancing rule
> among high-level consumer. Does it always guarantee that after several
> leadership change/temporary broker fail, it can always equally distribute
> the read among the brokers. Basically I think it's nice to have a API to
> let dev know which consumer reads from which broker otherwise I don't know
> anything behind the high-level consumer
>
> Thanks!
>
> Best,
> Siyuan
>

--
-- Guozhang

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB