Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> RE: Looks like consumer fetchers get stopped we are not getting any data


Copy link to this message
-
Re: Looks like consumer fetchers get stopped we are not getting any data
From your logs the channel with the brokers are broken, are the brokers
alive at that time?

Guozhang
On Fri, Jan 10, 2014 at 10:52 AM, Withers, Robert
<[EMAIL PROTECTED]>wrote:

> The core problem is our consumers stop consuming and lag increases.  We
> found this blog:
> https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-Myconsumerseemstohavestopped,why?.
>  This lists 3 possibilities.
>
> The blog also talks earlier about spurious rebalances, due to improper GC
> settings, but we couldn't find what GC settings to use.  We are considering
> changing the zookeeper timeouts.  We are a little confused about the
> various issues, the sequence of issues and what could cause the consumers
> to stop reading.  If the fetchers get shutdown, due to a
> ClosedByInterruptException in the "leader_finder" thread, which tells the
> "executor_watcher" thread to shutdown the fetchers, that would be another
> reason the consumers stop processing data.  Is this possible?
>
> Thank you,
> rob
>
> -----Original Message-----
> From: Seshadri, Balaji [mailto:[EMAIL PROTECTED]]
> Sent: Friday, January 10, 2014 11:40 AM
> To: [EMAIL PROTECTED]
> Subject: RE: Looks like consumer fetchers get stopped we are not getting
> any data
>
> It would be helpful if you guys can shed some light why all fetchers are
> getting stopped.
>
> -----Original Message-----
> From: Seshadri, Balaji [mailto:[EMAIL PROTECTED]]
> Sent: Friday, January 10, 2014 11:28 AM
> To: [EMAIL PROTECTED]
> Subject: RE: Looks like consumer fetchers get stopped we are not getting
> any data
>
> We also got the below error when this happens.
>
> {2014-01-10 00:58:11,292} INFO
>  [account-info-updated-hadoop-consumer_tm1mwdpl04-1389222553159-ad59660b_watcher_executor]
> (?:?) -
> [account-info-updated-hadoop-consumer_tm1mwdpl04-1389222553159-ad59660b],
> exception during rebalance
> org.I0Itec.zkclient.exception.ZkNoNodeException:
> org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode =
> NoNode for
> /consumers/account-info-updated-hadoop-consumer/ids/account-info-updated-hadoop-consumer_tm1mwdpl04-1389222553159-ad59660b
>         at
> org.I0Itec.zkclient.exception.ZkException.create(ZkException.java:47)
>         at
> org.I0Itec.zkclient.ZkClient.retryUntilConnected(ZkClient.java:685)
>         at org.I0Itec.zkclient.ZkClient.readData(ZkClient.java:766)
>         at org.I0Itec.zkclient.ZkClient.readData(ZkClient.java:761)
>         at kafka.utils.ZkUtils$.readData(Unknown Source)
>         at kafka.consumer.TopicCount$.constructTopicCount(Unknown Source)
>         at
> kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener.kafka$consumer$ZookeeperConsumerConnector$ZKRebalancerListener$$rebalance(Unknown
> Source)
>         at
> kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener$$anonfun$syncedRebalance$1.apply$mcVI$sp(Unknown
> Source)
>         at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:142)
>         at
> kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener.syncedRebalance(Unknown
> Source)
>         at
> kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener$$anon$1.run(Unknown
> Source) Caused by: org.apache.zookeeper.KeeperException$NoNodeException:
> KeeperErrorCode = NoNode for
> /consumers/account-info-updated-hadoop-consumer/ids/account-info-updated-hadoop-consumer_tm1mwdpl04-1389222553159-ad59660b
>         at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:102)
>         at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
>         at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:927)
>         at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:956)
>         at org.I0Itec.zkclient.ZkConnection.readData(ZkConnection.java:103)
>         at org.I0Itec.zkclient.ZkClient$9.call(ZkClient.java:770)
>         at org.I0Itec.zkclient.ZkClient$9.call(ZkClient.java:766)
>         at
> org.I0Itec.zkclient.ZkClient.retryUntilConnected(ZkClient.java:675)