Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Question about auto-rebalancing


Copy link to this message
-
Re: Question about auto-rebalancing
Hello Siyuan,

For the automatic leader re-election, yes we are considering to make it
work. Could you file a JIRA for this issue?

For the high-level consumer's rebalancing logic, you can find it at

https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-CanIpredicttheresultsoftheconsumerrebabalance%3F

Guozhang
On Fri, Oct 11, 2013 at 11:06 AM, [EMAIL PROTECTED] <[EMAIL PROTECTED]> wrote:

> Hi Jun,
>
> Thanks for your reply, but in a real cluster, one broker could serve
> different topics and different partitions, the simple consumer only has
> knowledge of brokers that are available but it has no knowledge to decide
> which broker is best to pick up to consume messages.  If you don't choose
> carefully, multiple simple consumer might end up with reading from same
> node which is definitely not good for performance.
> Interesting thing is I find out there is
> command kafka-preferred-replica-election.sh which will try to equally
> distribute the leadership among different brokers, this is good that I can
> always let my simple consumer reads from leader broker(even it fails, the
> replica will pick up as leader which is fine).  But why don't kafka cluster
> run this command automatically when there is a broker change(up/down) in
> the cluster so that the leadership can always be equally distributed among
> different brokers ASAP?  I think it's very good for simple consumer to
> decide which broker is good to read from.
>
> Another question is I'm also curious how high-level consumer is balanced. I
> assume each high-level consumer know other consumers(int the same group)
> which broker they read message from and it can try to avoid those brokers
> and to pick up a free one?  Is there a document for the balancing rule
> among high-level consumer. Does it always guarantee that after several
> leadership change/temporary broker fail, it can always equally distribute
> the read among the brokers. Basically I think it's nice to have a API to
> let dev know which consumer reads from which broker otherwise I don't know
> anything behind the high-level consumer
>
> Thanks!
>
> Best,
> Siyuan
>

--
-- Guozhang