Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka, mail # user - RE: Looks like consumer fetchers get stopped we are not getting any data


+
Withers, Robert 2014-01-10, 18:57
Copy link to this message
-
Re: Looks like consumer fetchers get stopped we are not getting any data
Guozhang Wang 2014-01-10, 19:28
From your logs the channel with the brokers are broken, are the brokers
alive at that time?

Guozhang
On Fri, Jan 10, 2014 at 10:52 AM, Withers, Robert
<[EMAIL PROTECTED]>wrote:

> The core problem is our consumers stop consuming and lag increases.  We
> found this blog:
> https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-Myconsumerseemstohavestopped,why?.
>  This lists 3 possibilities.
>
> The blog also talks earlier about spurious rebalances, due to improper GC
> settings, but we couldn't find what GC settings to use.  We are considering
> changing the zookeeper timeouts.  We are a little confused about the
> various issues, the sequence of issues and what could cause the consumers
> to stop reading.  If the fetchers get shutdown, due to a
> ClosedByInterruptException in the "leader_finder" thread, which tells the
> "executor_watcher" thread to shutdown the fetchers, that would be another
> reason the consumers stop processing data.  Is this possible?
>
> Thank you,
> rob
>
> -----Original Message-----
> From: Seshadri, Balaji [mailto:[EMAIL PROTECTED]]
> Sent: Friday, January 10, 2014 11:40 AM
> To: [EMAIL PROTECTED]
> Subject: RE: Looks like consumer fetchers get stopped we are not getting
> any data
>
> It would be helpful if you guys can shed some light why all fetchers are
> getting stopped.
>
> -----Original Message-----
> From: Seshadri, Balaji [mailto:[EMAIL PROTECTED]]
> Sent: Friday, January 10, 2014 11:28 AM
> To: [EMAIL PROTECTED]
> Subject: RE: Looks like consumer fetchers get stopped we are not getting
> any data
>
> We also got the below error when this happens.
>
> {2014-01-10 00:58:11,292} INFO
>  [account-info-updated-hadoop-consumer_tm1mwdpl04-1389222553159-ad59660b_watcher_executor]
> (?:?) -
> [account-info-updated-hadoop-consumer_tm1mwdpl04-1389222553159-ad59660b],
> exception during rebalance
> org.I0Itec.zkclient.exception.ZkNoNodeException:
> org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode =
> NoNode for
> /consumers/account-info-updated-hadoop-consumer/ids/account-info-updated-hadoop-consumer_tm1mwdpl04-1389222553159-ad59660b
>         at
> org.I0Itec.zkclient.exception.ZkException.create(ZkException.java:47)
>         at
> org.I0Itec.zkclient.ZkClient.retryUntilConnected(ZkClient.java:685)
>         at org.I0Itec.zkclient.ZkClient.readData(ZkClient.java:766)
>         at org.I0Itec.zkclient.ZkClient.readData(ZkClient.java:761)
>         at kafka.utils.ZkUtils$.readData(Unknown Source)
>         at kafka.consumer.TopicCount$.constructTopicCount(Unknown Source)
>         at
> kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener.kafka$consumer$ZookeeperConsumerConnector$ZKRebalancerListener$$rebalance(Unknown
> Source)
>         at
> kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener$$anonfun$syncedRebalance$1.apply$mcVI$sp(Unknown
> Source)
>         at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:142)
>         at
> kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener.syncedRebalance(Unknown
> Source)
>         at
> kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener$$anon$1.run(Unknown
> Source) Caused by: org.apache.zookeeper.KeeperException$NoNodeException:
> KeeperErrorCode = NoNode for
> /consumers/account-info-updated-hadoop-consumer/ids/account-info-updated-hadoop-consumer_tm1mwdpl04-1389222553159-ad59660b
>         at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:102)
>         at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
>         at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:927)
>         at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:956)
>         at org.I0Itec.zkclient.ZkConnection.readData(ZkConnection.java:103)
>         at org.I0Itec.zkclient.ZkClient$9.call(ZkClient.java:770)
>         at org.I0Itec.zkclient.ZkClient$9.call(ZkClient.java:766)
>         at
> org.I0Itec.zkclient.ZkClient.retryUntilConnected(ZkClient.java:675)

 
+
Guozhang Wang 2014-01-10, 19:30
+
Joel Koshy 2014-01-10, 23:37
+
Guozhang Wang 2014-01-11, 02:18
+
Guozhang Wang 2014-01-11, 22:54
+
Nanjegowda, Mithunraj 2014-01-11, 23:02
+
Jun Rao 2014-01-11, 05:12
+
Rob Withers 2014-01-11, 05:18
+
Guozhang Wang 2014-01-13, 03:21
+
Neha Narkhede 2014-01-13, 07:13