-Re: How does Kafka decide which Consumer out of multiple Consumer clients to assign to a single topic partition
Guozhang Wang 2013-08-22, 17:23
Thanks Neha for the clarification. I have created a new entry in FAQ for
Dennis, please let me know if this does not fully answer your question.
On Thu, Aug 22, 2013 at 9:29 AM, Neha Narkhede <[EMAIL PROTECTED]>wrote:
> We range partition a sorted list of topic-partitions over a sorted list of
> consumer instances. This makes the rebalancing algorithm deterministic. As
> soon as you bring up the 2nd consumer instance, if its position in the
> sorted list is before the position of the 1st consumer client, it will end
> up owning the partition.
> On Thu, Aug 22, 2013 at 8:53 AM, Dennis Haller <[EMAIL PROTECTED]
> > I have a situation where two high level consumers are being created to
> > consume a single topic. There is only one partition for the topic, so I
> > understand that only one Consumer will end up owning the topic and
> > receiving messages. The two consumers are created from two servers in a
> > redundant master - slave configuration, and it is our intention that the
> > servers should start in the same configuration predictably, with all the
> > Consumers active on the master server.
> > However, we find after both Consumers have been created that sometimes
> > first Consumer client succeeds in owning the topic and sometimes it is
> > second. I first thought that the first Consumer client to register with
> > the topic would be retained even if subsequent Consumers also register
> > the topic, but I see sometimes the second Consumer client succeeds in
> > replacing the first Consumer.
> > The logs show a rebalancing algorithm working after each Consumer is
> > registered.
> > In this case, where there is only one topic-partition, is it possible to
> > predict what Consumer client will own the topic? How is that rebalancing
> > done?
> > Thanks
> > Dennis