Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka, mail # user - Re: ConsumerRebalanceFailedException


Copy link to this message
-
Re: ConsumerRebalanceFailedException
Jun Rao 2013-03-13, 04:42
Do you see many rebalances in your consumer log?

Thanks,

Jun

On Tue, Mar 12, 2013 at 6:01 PM, Dragos Manolescu <
[EMAIL PROTECTED]> wrote:

> Hu Jun --
>
> I appreciate the fast turnaround. I use Zookeeper 3.4.5.
>
> I am not sure about the consumer GC: the exception happens when I fire up
> the client's VM which shouldn't be a busy time for the GC--I've been using
> autooffset.reset=largest so I assume that the client doesn't do a whole
> lot upon starting up. Perhaps I am wrong about that?
>
> Thanks,
>
> -Dragos
>
>
> On 3/12/13 11:53 AM, "Jun Rao" <[EMAIL PROTECTED]> wrote:
>
> >What's the ZK version? You should use 3.3.4 or above, which fixed some
> >bugs
> >that could cause rebalance to fail.
> >
> > Also, a typical reason for many rebalancing is the consumer side GC. If
> >so, you will see logs saying sth like "expired session" for ZK.
> >
> >Thanks,
> >
> >Jun
> >
> >
> >On Tue, Mar 12, 2013 at 9:47 AM, Dragos Manolescu <
> >[EMAIL PROTECTED]> wrote:
> >
> >> Hi --
> >>
> >> I have a Scala Kafka consumer, written in the image of the Java code
> >>from
> >> the Quickstart doc. I use the consumer for prototyping and debugging so
> >>I
> >> start and stop it quite frequently. I noticed that every now and then,
> >>upon
> >> starting the consumer, a ConsumerRebalanceFailedException is thrown (see
> >> below). What's causing the problem this exception is signaling and what
> >>are
> >> the recommended practices for avoiding it? I am using Kafka 0.7.2 that I
> >> built with Scala 2.9.2.
> >>
> >> Thanks,
> >>
> >> -Dragos
> >>
> >> Exception in thread "main"
> >>kafka.common.ConsumerRebalanceFailedException:
> >> Kconsumer-group_mseac02jh0rjdkq4-1363106041719-5b75c065 can't rebalance
> >> after 4 retries
> >> at
> >>
> >>kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener.syncedReba
> >>lance(ZookeeperConsumerConnector.scala:473)
> >> at
> >>
> >>kafka.consumer.ZookeeperConsumerConnector.kafka$consumer$ZookeeperConsume
> >>rConnector$$reinitializeConsumer(ZookeeperConsumerConnector.scala:733)
> >> at
> >>
> >>kafka.consumer.ZookeeperConsumerConnector.consume(ZookeeperConsumerConnec
> >>tor.scala:207)
> >> at
> >>
> >>kafka.consumer.ZookeeperConsumerConnector.createMessageStreams(ZookeeperC
> >>onsumerConnector.scala:137)
> >> at
> >>
> >>com.servicenow.syseng.kafka.KConsumer$delayedInit$body.apply(KConsumer.sc
> >>ala:69)
> >> at scala.Function0$class.apply$mcV$sp(Function0.scala:34)
> >> at
> >>scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12)
> >> at scala.App$$anonfun$main$1.apply(App.scala:60)
> >> at scala.App$$anonfun$main$1.apply(App.scala:60)
> >> at
> >>
> >>scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scal
> >>a:59)
> >> at scala.collection.immutable.List.foreach(List.scala:76)
> >> at
> >>
> >>scala.collection.generic.TraversableForwarder$class.foreach(TraversableFo
> >>rwarder.scala:30)
> >> at scala.App$class.main(App.scala:60)
> >>
> >>
> >>
>
>