Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> How does Kafka decide which Consumer out of multiple Consumer clients to assign to a single topic partition


Copy link to this message
-
Re: How does Kafka decide which Consumer out of multiple Consumer clients to assign to a single topic partition
Thanks Neha for the clarification. I have created a new entry in FAQ for
this question:

https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-CanIpredicttheresultsoftheconsumerrebabalance%3F

Dennis, please let me know if this does not fully answer your question.

Guozhang
On Thu, Aug 22, 2013 at 9:29 AM, Neha Narkhede <[EMAIL PROTECTED]>wrote:

> We range partition a sorted list of topic-partitions over a sorted list of
> consumer instances. This makes the rebalancing algorithm deterministic. As
> soon as you bring up the 2nd consumer instance, if its position in the
> sorted list is before the position of the 1st consumer client, it will end
> up owning the partition.
>
> Thanks,
> Neha
>
>
> On Thu, Aug 22, 2013 at 8:53 AM, Dennis Haller <[EMAIL PROTECTED]
> >wrote:
>
> > I have a situation where two high level consumers are being created to
> > consume a single topic. There is only one partition for the topic, so I
> > understand that only one Consumer will end up owning the topic and
> > receiving messages. The two consumers are created from two servers in a
> > redundant master - slave configuration, and it is our intention that the
> > servers should start in the same configuration predictably, with all the
> > Consumers active on the master server.
> >
> > However, we find after both Consumers have been created that sometimes
> the
> > first Consumer client succeeds in owning the topic and sometimes it is
> the
> > second. I first thought that the first Consumer  client to register with
> > the topic would be retained even if subsequent Consumers also register
> for
> > the topic, but I see sometimes the second Consumer client succeeds in
> > replacing the first Consumer.
> >
> > The logs show a rebalancing algorithm working after each Consumer is
> > registered.
> >
> > In this case, where there is only one topic-partition, is it possible to
> > predict what Consumer client will own the topic? How is that rebalancing
> > done?
> >
> > Thanks
> > Dennis
> >
>

--
-- Guozhang