Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Re: Looks like consumer fetchers get stopped we are not getting any data


Copy link to this message
-
Re: Looks like consumer fetchers get stopped we are not getting any data
Actually, the broken channel is broken by shutting down the
leader-finder-thread, which is shutdown either by a rebalance retry or
shutting down the consumer.

Do you see "begin rebalance ..." before this log entry? And if yes, search
to see if the rebalance keep failing.

Guozhang
On Fri, Jan 10, 2014 at 11:23 AM, Guozhang Wang <[EMAIL PROTECTED]> wrote:

> From your logs the channel with the brokers are broken, are the brokers
> alive at that time?
>
> Guozhang
>
>
> On Fri, Jan 10, 2014 at 10:52 AM, Withers, Robert <[EMAIL PROTECTED]
> > wrote:
>
>> The core problem is our consumers stop consuming and lag increases.  We
>> found this blog:
>> https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-Myconsumerseemstohavestopped,why?.
>>  This lists 3 possibilities.
>>
>> The blog also talks earlier about spurious rebalances, due to improper GC
>> settings, but we couldn't find what GC settings to use.  We are considering
>> changing the zookeeper timeouts.  We are a little confused about the
>> various issues, the sequence of issues and what could cause the consumers
>> to stop reading.  If the fetchers get shutdown, due to a
>> ClosedByInterruptException in the "leader_finder" thread, which tells the
>> "executor_watcher" thread to shutdown the fetchers, that would be another
>> reason the consumers stop processing data.  Is this possible?
>>
>> Thank you,
>> rob
>>
>> -----Original Message-----
>> From: Seshadri, Balaji [mailto:[EMAIL PROTECTED]]
>> Sent: Friday, January 10, 2014 11:40 AM
>> To: [EMAIL PROTECTED]
>> Subject: RE: Looks like consumer fetchers get stopped we are not getting
>> any data
>>
>> It would be helpful if you guys can shed some light why all fetchers are
>> getting stopped.
>>
>> -----Original Message-----
>> From: Seshadri, Balaji [mailto:[EMAIL PROTECTED]]
>> Sent: Friday, January 10, 2014 11:28 AM
>> To: [EMAIL PROTECTED]
>> Subject: RE: Looks like consumer fetchers get stopped we are not getting
>> any data
>>
>> We also got the below error when this happens.
>>
>> {2014-01-10 00:58:11,292} INFO
>>  [account-info-updated-hadoop-consumer_tm1mwdpl04-1389222553159-ad59660b_watcher_executor]
>> (?:?) -
>> [account-info-updated-hadoop-consumer_tm1mwdpl04-1389222553159-ad59660b],
>> exception during rebalance
>> org.I0Itec.zkclient.exception.ZkNoNodeException:
>> org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode =
>> NoNode for
>> /consumers/account-info-updated-hadoop-consumer/ids/account-info-updated-hadoop-consumer_tm1mwdpl04-1389222553159-ad59660b
>>         at
>> org.I0Itec.zkclient.exception.ZkException.create(ZkException.java:47)
>>         at
>> org.I0Itec.zkclient.ZkClient.retryUntilConnected(ZkClient.java:685)
>>         at org.I0Itec.zkclient.ZkClient.readData(ZkClient.java:766)
>>         at org.I0Itec.zkclient.ZkClient.readData(ZkClient.java:761)
>>         at kafka.utils.ZkUtils$.readData(Unknown Source)
>>         at kafka.consumer.TopicCount$.constructTopicCount(Unknown Source)
>>         at
>> kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener.kafka$consumer$ZookeeperConsumerConnector$ZKRebalancerListener$$rebalance(Unknown
>> Source)
>>         at
>> kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener$$anonfun$syncedRebalance$1.apply$mcVI$sp(Unknown
>> Source)
>>         at
>> scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:142)
>>         at
>> kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener.syncedRebalance(Unknown
>> Source)
>>         at
>> kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener$$anon$1.run(Unknown
>> Source) Caused by: org.apache.zookeeper.KeeperException$NoNodeException:
>> KeeperErrorCode = NoNode for
>> /consumers/account-info-updated-hadoop-consumer/ids/account-info-updated-hadoop-consumer_tm1mwdpl04-1389222553159-ad59660b
>>         at
>> org.apache.zookeeper.KeeperException.create(KeeperException.java:102)
>>         at
>> org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB