Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> consumer partition rebalancing

Copy link to this message
Re: consumer partition rebalancing
Actually an intermediate consumer would be useful for new topics, in which case it would probably need to watch the relevant zookeeper nodes. I could see a number of use cases where consumption strategies across a consumer group would deviate from the resident partition balancing algorithm. Perhaps this starts to get into Samza territory.
On Sep 11, 2013, at 1:14 PM, Joel Koshy <[EMAIL PROTECTED]> wrote:

> Correct - but since you wanted sticky allocation rebalancing wouldn't
> really be necessary.
> Thanks,
> Joel
> On Wed, Sep 11, 2013 at 10:08 AM, Kam Kasravi <[EMAIL PROTECTED]> wrote:
>> Thanks Joel. Just to be sure - SimpleConsumer (or an IntermediateConsumer as suggested below) does not join in partition rebalancing when another consumer in it's group joins or a new broker joins or a new topic is created?
>> ________________________________
>> From: Joel Koshy <[EMAIL PROTECTED]>
>> Sent: Tuesday, September 10, 2013 4:36 PM
>> Subject: Re: consumer partition rebalancing
>>>        * Assuming 1) theconsumerdoesn'tcontrolthe partition allocation within a topic and 2) theconstraintthat a single consumer C(i) within a consumer group C(g) must be the only reader of that partition:
>>>        * Are there ways to scale partition consumption if C(i) cannot keep up?
>> To some degree - by either reducing the number of streams that C(i) is
>> configured with; or by removing it altogether.
>>>        * A way to designate particular consumers should consume particular partitions based on consumer capability (similar to container allocation in YARN)?
>>>        * A way to designate sticky consumer / partition allocation so that a new consumer C(j) or new broker B(k) or new topic T(l) does not cause C(i) to lose particular partitions during rebalancing?
>> Right now, no. I think what you may be effectively suggesting is to
>> support some form of pluggable logic for partition ownership which
>> should work as long as it is consistent across all consumer instances.
>> However, a designated assignment list could be achieved by using
>> SimpleConsumer - although you would then need to write a little code
>> to react to leader changes. (We currently don't have a
>> intermediate-level consumer - i.e., a consumer that can be designated
>> to consume from a given partition and automatically react to leader
>> changes.)
>> Joel