Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Consumer stops consuming after 5 gb


Copy link to this message
-
Re: Consumer stops consuming after 5 gb
Not sure about the error. However, your consumer seems to be lagging. Are
the consumer offsets moving at all? If not, are your consumer threads still
alive? You probably want to try/catch the consumer code to see if there is
any unexpected exceptions.

Thanks,

Jun
On Tue, Jul 23, 2013 at 7:58 AM, Anurup Raveendran <
[EMAIL PROTECTED]> wrote:

> After consuming about 5 gb of messages , it stopped consuming and it got
> stuck at a particular offset. After running the following command I got an
> error regarding the broker.
>
>
>
>
> $ bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --group
> air-DummyProductionConsumerGroup-After --zkconnect 172.16.18.65:2181--topic
> air
>
>
>
> [2013-07-23 20:19:42,748] INFO Starting ZkClient event thread.
> (org.I0Itec.zkclient.ZkEventThread)
> [2013-07-23 20:19:42,757] INFO Client
> environment:zookeeper.version=3.3.3-1203054, built on 11/17/2011 05:47 GMT
> (org.apache.zookeeper.ZooKeeper)
> [2013-07-23 20:19:42,757] INFO Client environment:host.name=localhost
> (org.apache.zookeeper.ZooKeeper)
> [2013-07-23 20:19:42,757] INFO Client environment:java.version=1.7.0_21
> (org.apache.zookeeper.ZooKeeper)
> [2013-07-23 20:19:42,757] INFO Client environment:java.vendor=Oracle
> Corporation (org.apache.zookeeper.ZooKeeper)
> [2013-07-23 20:19:42,757] INFO Client
> environment:java.home=/usr/java/jdk1.7.0_21/jre
> (org.apache.zookeeper.ZooKeeper)
> [2013-07-23 20:19:42,757] INFO Client
>
> environment:java.class.path=:bin/../project/boot/scala-2.8.0/lib/scala-compiler.jar:bin/../project/boot/scala-2.8.0/lib/scala-library.jar:bin/../core/target/scala_2.8.0/kafka-0.7.2.jar:bin/../core/lib/*.jar:bin/../perf/target/scala_2.8.0/kafka-perf-0.7.2.jar:bin/../core/lib_managed/scala_2.8.0/compile/jopt-simple-3.2.jar:bin/../core/lib_managed/scala_2.8.0/compile/log4j-1.2.15.jar:bin/../core/lib_managed/scala_2.8.0/compile/snappy-java-1.0.4.1.jar:bin/../core/lib_managed/scala_2.8.0/compile/zkclient-0.1.jar:bin/../core/lib_managed/scala_2.8.0/compile/zookeeper-3.3.4.jar
> (org.apache.zookeeper.ZooKeeper)
> [2013-07-23 20:19:42,758] INFO Client
>
> environment:java.library.path=/usr/local/oracle:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
> (org.apache.zookeeper.ZooKeeper)
> [2013-07-23 20:19:42,758] INFO Client environment:java.io.tmpdir=/tmp
> (org.apache.zookeeper.ZooKeeper)
> [2013-07-23 20:19:42,758] INFO Client environment:java.compiler=<NA>
> (org.apache.zookeeper.ZooKeeper)
> [2013-07-23 20:19:42,758] INFO Client environment:os.name=Linux
> (org.apache.zookeeper.ZooKeeper)
> [2013-07-23 20:19:42,758] INFO Client environment:os.arch=amd64
> (org.apache.zookeeper.ZooKeeper)
> [2013-07-23 20:19:42,758] INFO Client
> environment:os.version=2.6.32-279.22.1.el6.x86_64
> (org.apache.zookeeper.ZooKeeper)
> [2013-07-23 20:19:42,758] INFO Client environment:user.name=cleartrip
> (org.apache.zookeeper.ZooKeeper)
> [2013-07-23 20:19:42,758] INFO Client environment:user.home=/home/cleartrip
> (org.apache.zookeeper.ZooKeeper)
> [2013-07-23 20:19:42,758] INFO Client
> environment:user.dir=/home/cleartrip/kafka/kafka-0.7.2-incubating-src
> (org.apache.zookeeper.ZooKeeper)
> [2013-07-23 20:19:42,759] INFO Initiating client connection, connectString=
> 172.16.18.65:2181 sessionTimeout=30000
> watcher=org.I0Itec.zkclient.ZkClient@6926e27
> (org.apache.zookeeper.ZooKeeper)
> [2013-07-23 20:19:42,783] INFO Opening socket connection to server /
> 172.16.18.65:2181 (org.apache.zookeeper.ClientCnxn)
> [2013-07-23 20:19:42,788] INFO Socket connection established to zoo1/
> 172.16.18.65:2181, initiating session (org.apache.zookeeper.ClientCnxn)
> [2013-07-23 20:19:42,800] INFO Session establishment complete on server
> zoo1/172.16.18.65:2181, sessionid = 0x13fc7235423015f, negotiated timeout
> =
> 30000 (org.apache.zookeeper.ClientCnxn)
> [2013-07-23 20:19:42,802] INFO zookeeper state changed (SyncConnected)
> (org.I0Itec.zkclient.ZkClient)
> air-DummyProductionConsumerGroup-After,air,1-0