Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Re: Question about Split brain problem for the case of a single consumer per topic


Copy link to this message
-
Re: Question about Split brain problem for the case of a single consumer per topic
Hi Itai,

It will be easier to explain things if we know your use case. I'll
take a stab at your questions -

1.      At most one consumer per topic.
2.      That this single consumer would consume all partitions.

If you have one consumer in a group, you can achieve this. However, I
still wonder why you would want this guarantee since essentially that
consumer is a single point of failure.

3.      In case of consumer failure, eventual guarantee of one
consumer per topic.

If you deploy just one consumer in a group, it is your responsibility
to handle its failover. Kafka will only handle automatic failovers for
consumers in the same group.

4.      In case of consumer recovery, some way to rebalance the topics
across consumers.

If those consumers are in the same group, rebalance will happen automatically.

Thanks,
Neha

On Thu, Apr 11, 2013 at 7:51 PM, Itai Frenkel <[EMAIL PROTECTED]> wrote:
> Hello,
>
> I am evaluating the usage of Kafka, and read the "Split brain problem: The consumer rebalancing design suffers from a split brain problem" in
> https://cwiki.apache.org/KAFKA/consumer-co-ordinator.html
>
> My question is this - assuming I would like to have a single consumer per topic, would I still suffer from the Split Brain problem?
>
> Can the existing algorithm be configured to guarantee that:
>
> 1.      At most one consumer per topic.
>
> 2.      That this single consumer would consume all partitions.
>
> 3.      In case of consumer failure, eventual guarantee of one consumer per topic.
>
> 4.      In case of consumer recovery, some way to rebalance the topics across consumers.
>
>
> Thanks,
> Itai
>

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB