Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Re: ConsumerRebalanceFailedException


Copy link to this message
-
Re: ConsumerRebalanceFailedException
One alternative method is to check the zookeeper consumer registration
path, if the node is gone then try to restart the consumer after the
sesstion timeout.

Guozhang
On Mon, Dec 30, 2013 at 7:56 PM, Hanish Bansal <
[EMAIL PROTECTED]> wrote:

> As default zookeeper.session.timeout.ms is 6000 and i look into the
> details
> this value is negotiable.  We try to set this value to less than 4000 to
> expire the session early but it is negotiated by zookeeper and set to 4000
> ms.
>
> We have a backend script running which check in each second that if
> consumer service is not running then start it. So using this we are
> starting consumer service within second without any wait.
> *"connector.shutdown()"
> is good option for this but that will not work if consumer is killed
> abnormally using kill -9*.
>
> Other option i am seeing to put "Thread.sleep(sessionTimeoutMilliseconds)"
> in consumer service before start but that is also not good  option.
>
> *When ConsumerRebalanceFailedException occurs then it stop consumes the
> data. But expected behaviour should be like this : If C**onsumerRe*
> *balanceFai**ledExcepti**on occurs due to zookeeper session then it should
> wait for that timeout interval. If previous session is timeout, it should
> reconnect and start consuming the data.*
>
> Any other way to handle it?
>
> Also i want to know what is suggested value  for
> zookeeper.session.timeout.ms in production ?
>
>
> On Mon, Dec 30, 2013 at 11:49 PM, Guozhang Wang <[EMAIL PROTECTED]>
> wrote:
>
> > Yes, it applies to consumer too.
> >
> >
> > On Mon, Dec 30, 2013 at 11:46 AM, Yu, Libo <[EMAIL PROTECTED]> wrote:
> >
> > > Hi Jun,
> > >
> > > zookeeper.session.timeout.ms is used in a broker's configuration and
> > > manages brokers' registration with zk.
> > > Does it apply to consumer as well? Thanks.
> > >
> > > Regards,
> > >
> > > Libo
> > >
> > >
> > > -----Original Message-----
> > > From: Jun Rao [mailto:[EMAIL PROTECTED]]
> > > Sent: Monday, December 30, 2013 11:13 AM
> > > To: [EMAIL PROTECTED]
> > > Subject: Re: ConsumerRebalanceFailedException
> > >
> > > If the consumer is not shut down properly, it will take
> > > zookeeper.session.timeout.ms before the consumer is deregistered from
> > ZK.
> > > If you restart the consumer before that, rebalances may fail.
> > >
> > > Make sure that you call connector.shutdown() when you shut down the
> > > consumer
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > >
> > > On Mon, Dec 30, 2013 at 1:58 AM, Hanish Bansal <
> > > [EMAIL PROTECTED]> wrote:
> > >
> > > > Hi All,
> > > >
> > > > I am getting consumer rebalance failed exception if i restart my
> > > > consumer within 1-3 seconds.
> > > >
> > > > Exception trace is:
> > > >
> > > > Caused by: kafka.common.ConsumerRebalanceFailedException:
> > > > indexConsumerGroup1_IMPETUS-I0027C-1388416992091-ac0d82d7 can't
> > > > rebalance after 4 retries
> > > >     at
> > > >
> > > >
> kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener.syncedR
> > > > ebalance(Unknown
> > > > Source)
> > > >     at
> > > >
> > > >
> kafka.consumer.ZookeeperConsumerConnector.kafka$consumer$ZookeeperCons
> > > > umerConnector$$reinitializeConsumer(Unknown
> > > > Source)
> > > >     at kafka.consumer.ZookeeperConsumerConnector.consume(Unknown
> > Source)
> > > >     at
> > > >
> > > >
> kafka.javaapi.consumer.ZookeeperConsumerConnector.createMessageStreams
> > > > (Unknown
> > > > Source)
> > > >     at
> > > >
> > > >
> kafka.javaapi.consumer.ZookeeperConsumerConnector.createMessageStreams
> > > > (Unknown
> > > > Source)
> > > >
> > > >
> > > > Is this exception depends on any of below properties:
> > > > zookeeper.session.timeout.ms  6000
> > > >  zookeeper.connection.timeout.ms 6000
> > > >
> > > > If i kill the consumer and start that again after 5-6 sec then it
> > > > started working properly without throwing any exception.
> > > >
> > > > If i start consumer immediately after killing that then
> > > > ConsumerRebalanceFailedException occurs.
 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB