Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> ConsumerRebalanceFailedException when broker unavailable


Copy link to this message
-
Re: ConsumerRebalanceFailedException when broker unavailable
I think this is addressed in KAFKA-969

On Fri, Jul 12, 2013 at 5:55 PM, Florin Trofin <[EMAIL PROTECTED]> wrote:
> Hello,
>
> I'm going through a series of tests with Kafka 0.8 to see what the failure cases are. One simple scenario is this:
>
> 1. Launch Zookeeper
> 2. Launch my consumer (written in scala) but don't launch a Kafka broker (I am not using replication in my configuration).
>
> What happens in this particular scenario is that my consumer hits the createMessageStreams which internally makes 4 failed attempts and then it raises a ConsumerRebalanceFailedException:
>
> kafka.common.ConsumerRebalanceFailedException: zfilter_vm-florin-ubuntu-d-12-1373675347482-ff93b841 can't rebalance after 4 retries
>         at kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener.syncedRebalance(ZookeeperConsumerConnector.scala:397)
>         at kafka.consumer.ZookeeperConsumerConnector.kafka$consumer$ZookeeperConsumerConnector$$reinitializeConsumer(ZookeeperConsumerConnector.scala:681)
>         at kafka.consumer.ZookeeperConsumerConnector.consume(ZookeeperConsumerConnector.scala:205)
>         at kafka.consumer.ZookeeperConsumerConnector.createMessageStreams(ZookeeperConsumerConnector.scala:133)
>         at kafka.consumer.ZookeeperConsumerConnector.createMessageStreams(ZookeeperConsumerConnector.scala:126)
>
>
> I was expecting a more specific exception - something like NoBrokerAvailable instead of this. I can add a catch clause for this exception and retry the operation in a loop until the broker comes online but I dislike the approach because I am making an assumption here - that ConsumerRebalanceFailedException means "no broker available". Is there a better way to do this? Do you have a pattern for consumers for when brokers go offline and come back up?
>
> Thanks a lot!
>
> Florin

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB