-Re: consumer partition rebalancing
Kam Kasravi 2013-09-11, 22:01
Actually an intermediate consumer would be useful for new topics, in which case it would probably need to watch the relevant zookeeper nodes. I could see a number of use cases where consumption strategies across a consumer group would deviate from the resident partition balancing algorithm. Perhaps this starts to get into Samza territory.
On Sep 11, 2013, at 1:14 PM, Joel Koshy <[EMAIL PROTECTED]> wrote:
> Correct - but since you wanted sticky allocation rebalancing wouldn't
> really be necessary.
> On Wed, Sep 11, 2013 at 10:08 AM, Kam Kasravi <[EMAIL PROTECTED]> wrote:
>> Thanks Joel. Just to be sure - SimpleConsumer (or an IntermediateConsumer as suggested below) does not join in partition rebalancing when another consumer in it's group joins or a new broker joins or a new topic is created?
>> From: Joel Koshy <[EMAIL PROTECTED]>
>> To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>; Kam Kasravi <[EMAIL PROTECTED]>
>> Sent: Tuesday, September 10, 2013 4:36 PM
>> Subject: Re: consumer partition rebalancing
>>> * Assuming 1) theconsumerdoesn'tcontrolthe partition allocation within a topic and 2) theconstraintthat a single consumer C(i) within a consumer group C(g) must be the only reader of that partition:
>>> * Are there ways to scale partition consumption if C(i) cannot keep up?
>> To some degree - by either reducing the number of streams that C(i) is
>> configured with; or by removing it altogether.
>>> * A way to designate particular consumers should consume particular partitions based on consumer capability (similar to container allocation in YARN)?
>>> * A way to designate sticky consumer / partition allocation so that a new consumer C(j) or new broker B(k) or new topic T(l) does not cause C(i) to lose particular partitions during rebalancing?
>> Right now, no. I think what you may be effectively suggesting is to
>> support some form of pluggable logic for partition ownership which
>> should work as long as it is consistent across all consumer instances.
>> However, a designated assignment list could be achieved by using
>> SimpleConsumer - although you would then need to write a little code
>> to react to leader changes. (We currently don't have a
>> intermediate-level consumer - i.e., a consumer that can be designated
>> to consume from a given partition and automatically react to leader