Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Re: Looks like consumer fetchers get stopped we are not getting any data


Copy link to this message
-
Re: Looks like consumer fetchers get stopped we are not getting any data
If a consumer rebalances for any reason (e.g., if a consumer
in the group has a soft failure such as a long GC) then the fetchers
are stopped as part of the rebalance process. The sequence is as
follows:

- Stop fetchers
- Commit offsets
- Release partition ownership
- Rebalance (i.e., figure out what partitions this consumer should now
  consume with the updated set of consumers)
- Acquire partition ownership
- Add fetchers to those partitions and resume consumption

i.e., rebalances should complete successfully and fetching should
resume. If you have any rebalance failures (search for "can't
rebalance after") then the consumer will effectively stop.

From later in this thread it seems your consumer somehow got into a
weird state in zookeeper, so your only recourse at this point may be
to stop all your consumers and restart.

Thanks,

Joel

> If the fetchers get shutdown, due to a ClosedByInterruptException in the "leader_finder" thread, which tells the "executor_watcher" thread to shutdown the fetchers, that would be another reason the consumers stop processing data.  Is this possible?
>

>
> -----Original Message-----
> From: Seshadri, Balaji [mailto:[EMAIL PROTECTED]]
> Sent: Friday, January 10, 2014 11:40 AM
> To: [EMAIL PROTECTED]
> Subject: RE: Looks like consumer fetchers get stopped we are not getting any data
>
> It would be helpful if you guys can shed some light why all fetchers are getting stopped.
>
> -----Original Message-----
> From: Seshadri, Balaji [mailto:[EMAIL PROTECTED]]
> Sent: Friday, January 10, 2014 11:28 AM
> To: [EMAIL PROTECTED]
> Subject: RE: Looks like consumer fetchers get stopped we are not getting any data
>
> We also got the below error when this happens.
>
> {2014-01-10 00:58:11,292} INFO  [account-info-updated-hadoop-consumer_tm1mwdpl04-1389222553159-ad59660b_watcher_executor] (?:?) - [account-info-updated-hadoop-consumer_tm1mwdpl04-1389222553159-ad59660b], exception during rebalance
> org.I0Itec.zkclient.exception.ZkNoNodeException: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /consumers/account-info-updated-hadoop-consumer/ids/account-info-updated-hadoop-consumer_tm1mwdpl04-1389222553159-ad59660b
>         at org.I0Itec.zkclient.exception.ZkException.create(ZkException.java:47)
>         at org.I0Itec.zkclient.ZkClient.retryUntilConnected(ZkClient.java:685)
>         at org.I0Itec.zkclient.ZkClient.readData(ZkClient.java:766)
>         at org.I0Itec.zkclient.ZkClient.readData(ZkClient.java:761)
>         at kafka.utils.ZkUtils$.readData(Unknown Source)
>         at kafka.consumer.TopicCount$.constructTopicCount(Unknown Source)
>         at kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener.kafka$consumer$ZookeeperConsumerConnector$ZKRebalancerListener$$rebalance(Unknown Source)
>         at kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener$$anonfun$syncedRebalance$1.apply$mcVI$sp(Unknown Source)
>         at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:142)
>         at kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener.syncedRebalance(Unknown Source)
>         at kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener$$anon$1.run(Unknown Source) Caused by: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /consumers/account-info-updated-hadoop-consumer/ids/account-info-updated-hadoop-consumer_tm1mwdpl04-1389222553159-ad59660b
>         at org.apache.zookeeper.KeeperException.create(KeeperException.java:102)
>         at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
>         at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:927)
>         at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:956)
>         at org.I0Itec.zkclient.ZkConnection.readData(ZkConnection.java:103)
>         at org.I0Itec.zkclient.ZkClient$9.call(ZkClient.java:770)
>         at org.I0Itec.zkclient.ZkClient$9.call(ZkClient.java:766)
 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB