Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka >> mail # user >> How does Kafka decide which Consumer out of multiple Consumer clients to assign to a single topic partition


+
Dennis Haller 2013-08-22, 15:54
+
Neha Narkhede 2013-08-22, 16:30
Copy link to this message
-
Re: How does Kafka decide which Consumer out of multiple Consumer clients to assign to a single topic partition
Thanks Neha for the clarification. I have created a new entry in FAQ for
this question:

https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-CanIpredicttheresultsoftheconsumerrebabalance%3F

Dennis, please let me know if this does not fully answer your question.

Guozhang
On Thu, Aug 22, 2013 at 9:29 AM, Neha Narkhede <[EMAIL PROTECTED]>wrote:

> We range partition a sorted list of topic-partitions over a sorted list of
> consumer instances. This makes the rebalancing algorithm deterministic. As
> soon as you bring up the 2nd consumer instance, if its position in the
> sorted list is before the position of the 1st consumer client, it will end
> up owning the partition.
>
> Thanks,
> Neha
>
>
> On Thu, Aug 22, 2013 at 8:53 AM, Dennis Haller <[EMAIL PROTECTED]
> >wrote:
>
> > I have a situation where two high level consumers are being created to
> > consume a single topic. There is only one partition for the topic, so I
> > understand that only one Consumer will end up owning the topic and
> > receiving messages. The two consumers are created from two servers in a
> > redundant master - slave configuration, and it is our intention that the
> > servers should start in the same configuration predictably, with all the
> > Consumers active on the master server.
> >
> > However, we find after both Consumers have been created that sometimes
> the
> > first Consumer client succeeds in owning the topic and sometimes it is
> the
> > second. I first thought that the first Consumer  client to register with
> > the topic would be retained even if subsequent Consumers also register
> for
> > the topic, but I see sometimes the second Consumer client succeeds in
> > replacing the first Consumer.
> >
> > The logs show a rebalancing algorithm working after each Consumer is
> > registered.
> >
> > In this case, where there is only one topic-partition, is it possible to
> > predict what Consumer client will own the topic? How is that rebalancing
> > done?
> >
> > Thanks
> > Dennis
> >
>

--
-- Guozhang

 
+
Dennis Haller 2013-08-22, 17:46
+
Guozhang Wang 2013-08-22, 16:32
+
Dennis Haller 2013-08-22, 17:45
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB