Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka >> mail # user >> RE: Looks like consumer fetchers get stopped we are not getting any data


+
Withers, Robert 2014-01-10, 18:57
+
Guozhang Wang 2014-01-10, 19:28
+
Guozhang Wang 2014-01-10, 19:30
+
Joel Koshy 2014-01-10, 23:37
Copy link to this message
-
Re: Looks like consumer fetchers get stopped we are not getting any data
From the logs it seems the consumer 562b6738's registry node in Zookeeper
has lost:

NoNode for
/consumers/account-activated-hadoop-consumer/ids/account-activated-hadoop-consumer_tm1mwdpl04-1389222557906-562b6738

As Joel suggested for now you may just stop all your consumers and restart,
to debug you may need to investigate into zookeeper's log checking is there
any session expiration or close socket events happened which cause ZK to
delete the registry node.

Which Kafka version are you using?

Guozhang
On Fri, Jan 10, 2014 at 3:36 PM, Joel Koshy <[EMAIL PROTECTED]> wrote:

> If a consumer rebalances for any reason (e.g., if a consumer
> in the group has a soft failure such as a long GC) then the fetchers
> are stopped as part of the rebalance process. The sequence is as
> follows:
>
> - Stop fetchers
> - Commit offsets
> - Release partition ownership
> - Rebalance (i.e., figure out what partitions this consumer should now
>   consume with the updated set of consumers)
> - Acquire partition ownership
> - Add fetchers to those partitions and resume consumption
>
> i.e., rebalances should complete successfully and fetching should
> resume. If you have any rebalance failures (search for "can't
> rebalance after") then the consumer will effectively stop.
>
> From later in this thread it seems your consumer somehow got into a
> weird state in zookeeper, so your only recourse at this point may be
> to stop all your consumers and restart.
>
> Thanks,
>
> Joel
>
> > If the fetchers get shutdown, due to a ClosedByInterruptException in the
> "leader_finder" thread, which tells the "executor_watcher" thread to
> shutdown the fetchers, that would be another reason the consumers stop
> processing data.  Is this possible?
> >
>
> >
> > -----Original Message-----
> > From: Seshadri, Balaji [mailto:[EMAIL PROTECTED]]
> > Sent: Friday, January 10, 2014 11:40 AM
> > To: [EMAIL PROTECTED]
> > Subject: RE: Looks like consumer fetchers get stopped we are not getting
> any data
> >
> > It would be helpful if you guys can shed some light why all fetchers are
> getting stopped.
> >
> > -----Original Message-----
> > From: Seshadri, Balaji [mailto:[EMAIL PROTECTED]]
> > Sent: Friday, January 10, 2014 11:28 AM
> > To: [EMAIL PROTECTED]
> > Subject: RE: Looks like consumer fetchers get stopped we are not getting
> any data
> >
> > We also got the below error when this happens.
> >
> > {2014-01-10 00:58:11,292} INFO
>  [account-info-updated-hadoop-consumer_tm1mwdpl04-1389222553159-ad59660b_watcher_executor]
> (?:?) -
> [account-info-updated-hadoop-consumer_tm1mwdpl04-1389222553159-ad59660b],
> exception during rebalance
> > org.I0Itec.zkclient.exception.ZkNoNodeException:
> org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode =
> NoNode for
> /consumers/account-info-updated-hadoop-consumer/ids/account-info-updated-hadoop-consumer_tm1mwdpl04-1389222553159-ad59660b
> >         at
> org.I0Itec.zkclient.exception.ZkException.create(ZkException.java:47)
> >         at
> org.I0Itec.zkclient.ZkClient.retryUntilConnected(ZkClient.java:685)
> >         at org.I0Itec.zkclient.ZkClient.readData(ZkClient.java:766)
> >         at org.I0Itec.zkclient.ZkClient.readData(ZkClient.java:761)
> >         at kafka.utils.ZkUtils$.readData(Unknown Source)
> >         at kafka.consumer.TopicCount$.constructTopicCount(Unknown Source)
> >         at
> kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener.kafka$consumer$ZookeeperConsumerConnector$ZKRebalancerListener$$rebalance(Unknown
> Source)
> >         at
> kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener$$anonfun$syncedRebalance$1.apply$mcVI$sp(Unknown
> Source)
> >         at
> scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:142)
> >         at
> kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener.syncedRebalance(Unknown
> Source)
> >         at
> kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener$$anon$1.run(Unknown
> Source) Caused by: org.apache.zookeeper.KeeperException$NoNodeException:

 
+
Guozhang Wang 2014-01-11, 22:54
+
Nanjegowda, Mithunraj 2014-01-11, 23:02
+
Jun Rao 2014-01-11, 05:12
+
Rob Withers 2014-01-11, 05:18
+
Guozhang Wang 2014-01-13, 03:21
+
Neha Narkhede 2014-01-13, 07:13