I have a Scala Kafka consumer, written in the image of the Java code from the Quickstart doc. I use the consumer for prototyping and debugging so I start and stop it quite frequently. I noticed that every now and then, upon starting the consumer, a ConsumerRebalanceFailedException is thrown (see below). What's causing the problem this exception is signaling and what are the recommended practices for avoiding it? I am using Kafka 0.7.2 that I built with Scala 2.9.2.
Exception in thread "main" kafka.common.ConsumerRebalanceFailedException: Kconsumer-group_mseac02jh0rjdkq4-1363106041719-5b75c065 can't rebalance after 4 retries at kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener.syncedRebalance(ZookeeperConsumerConnector.scala:473) at kafka.consumer.ZookeeperConsumerConnector.kafka$consumer$ZookeeperConsumerConnector$$reinitializeConsumer(ZookeeperConsumerConnector.scala:733) at kafka.consumer.ZookeeperConsumerConnector.consume(ZookeeperConsumerConnector.scala:207) at kafka.consumer.ZookeeperConsumerConnector.createMessageStreams(ZookeeperConsumerConnector.scala:137) at com.servicenow.syseng.kafka.KConsumer$delayedInit$body.apply(KConsumer.scala:69) at scala.Function0$class.apply$mcV$sp(Function0.scala:34) at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12) at scala.App$$anonfun$main$1.apply(App.scala:60) at scala.App$$anonfun$main$1.apply(App.scala:60) at scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:59) at scala.collection.immutable.List.foreach(List.scala:76) at scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:30) at scala.App$class.main(App.scala:60)
I appreciate the fast turnaround. I use Zookeeper 3.4.5.
I am not sure about the consumer GC: the exception happens when I fire up the client's VM which shouldn't be a busy time for the GC--I've been using autooffset.reset=largest so I assume that the client doesn't do a whole lot upon starting up. Perhaps I am wrong about that?
-Dragos On 3/12/13 11:53 AM, "Jun Rao" <[EMAIL PROTECTED]> wrote:
I have seen those exceptions when you restart your Kafka client before the configured Zookeeper session timeout has occured. Try and wait a bit longer between client restarts (we wait for 40s) and see if those exceptions go away.
On Tuesday, March 12, 2013, Dragos Manolescu wrote:
NEW: Monitor These Apps!
Apache Lucene, Apache Solr and all other Apache Software Foundation project and their respective logos are trademarks of the Apache Software Foundation.
Elasticsearch, Kibana, Logstash, and Beats are trademarks of Elasticsearch BV, registered in the U.S. and in other countries. This site and Sematext Group is in no way affiliated with Elasticsearch BV.
Service operated by Sematext