Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Re: ConsumerRebalanceFailedException


Copy link to this message
-
Re: ConsumerRebalanceFailedException
Do you see many rebalances in your consumer log?

Thanks,

Jun

On Tue, Mar 12, 2013 at 6:01 PM, Dragos Manolescu <
[EMAIL PROTECTED]> wrote:

> Hu Jun --
>
> I appreciate the fast turnaround. I use Zookeeper 3.4.5.
>
> I am not sure about the consumer GC: the exception happens when I fire up
> the client's VM which shouldn't be a busy time for the GC--I've been using
> autooffset.reset=largest so I assume that the client doesn't do a whole
> lot upon starting up. Perhaps I am wrong about that?
>
> Thanks,
>
> -Dragos
>
>
> On 3/12/13 11:53 AM, "Jun Rao" <[EMAIL PROTECTED]> wrote:
>
> >What's the ZK version? You should use 3.3.4 or above, which fixed some
> >bugs
> >that could cause rebalance to fail.
> >
> > Also, a typical reason for many rebalancing is the consumer side GC. If
> >so, you will see logs saying sth like "expired session" for ZK.
> >
> >Thanks,
> >
> >Jun
> >
> >
> >On Tue, Mar 12, 2013 at 9:47 AM, Dragos Manolescu <
> >[EMAIL PROTECTED]> wrote:
> >
> >> Hi --
> >>
> >> I have a Scala Kafka consumer, written in the image of the Java code
> >>from
> >> the Quickstart doc. I use the consumer for prototyping and debugging so
> >>I
> >> start and stop it quite frequently. I noticed that every now and then,
> >>upon
> >> starting the consumer, a ConsumerRebalanceFailedException is thrown (see
> >> below). What's causing the problem this exception is signaling and what
> >>are
> >> the recommended practices for avoiding it? I am using Kafka 0.7.2 that I
> >> built with Scala 2.9.2.
> >>
> >> Thanks,
> >>
> >> -Dragos
> >>
> >> Exception in thread "main"
> >>kafka.common.ConsumerRebalanceFailedException:
> >> Kconsumer-group_mseac02jh0rjdkq4-1363106041719-5b75c065 can't rebalance
> >> after 4 retries
> >> at
> >>
> >>kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener.syncedReba
> >>lance(ZookeeperConsumerConnector.scala:473)
> >> at
> >>
> >>kafka.consumer.ZookeeperConsumerConnector.kafka$consumer$ZookeeperConsume
> >>rConnector$$reinitializeConsumer(ZookeeperConsumerConnector.scala:733)
> >> at
> >>
> >>kafka.consumer.ZookeeperConsumerConnector.consume(ZookeeperConsumerConnec
> >>tor.scala:207)
> >> at
> >>
> >>kafka.consumer.ZookeeperConsumerConnector.createMessageStreams(ZookeeperC
> >>onsumerConnector.scala:137)
> >> at
> >>
> >>com.servicenow.syseng.kafka.KConsumer$delayedInit$body.apply(KConsumer.sc
> >>ala:69)
> >> at scala.Function0$class.apply$mcV$sp(Function0.scala:34)
> >> at
> >>scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12)
> >> at scala.App$$anonfun$main$1.apply(App.scala:60)
> >> at scala.App$$anonfun$main$1.apply(App.scala:60)
> >> at
> >>
> >>scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scal
> >>a:59)
> >> at scala.collection.immutable.List.foreach(List.scala:76)
> >> at
> >>
> >>scala.collection.generic.TraversableForwarder$class.foreach(TraversableFo
> >>rwarder.scala:30)
> >> at scala.App$class.main(App.scala:60)
> >>
> >>
> >>
>
>

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB