Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka >> mail # user >> Consumer stops consuming after 5 gb


+
Anurup Raveendran 2013-07-23, 14:59
+
Jun Rao 2013-07-23, 15:37
+
Samir Madhavan 2013-07-23, 16:56
+
Samir Madhavan 2013-07-23, 17:18
Copy link to this message
-
Re: Consumer stops consuming after 5 gb
Samir,

When you observe that the consumer hangs, could you send around the thread
dump. Also can you check the value of the FetchRate mbean ?

Thanks,
Neha
On Tue, Jul 23, 2013 at 10:17 AM, Samir Madhavan <
[EMAIL PROTECTED]> wrote:

> We just observed that the consumer is occupying almost complete RAM which
> is 4Gb.
>
>
> On Tue, Jul 23, 2013 at 10:25 PM, Samir Madhavan <
> [EMAIL PROTECTED]> wrote:
>
> > One of the common pattern we observed when we ran consumer from start
> > multiple times is that it catches up till the data that producer has
> > produced and then it just gets hung. If we kill it and then start it
> again,
> > it starts consuming and then again it just gets hung. We have added the
> try
> > catch statements but there seems to be no error coming.
> >
> >
> > On Tue, Jul 23, 2013 at 9:06 PM, Jun Rao <[EMAIL PROTECTED]> wrote:
> >
> >> Not sure about the error. However, your consumer seems to be lagging.
> Are
> >> the consumer offsets moving at all? If not, are your consumer threads
> >> still
> >> alive? You probably want to try/catch the consumer code to see if there
> is
> >> any unexpected exceptions.
> >>
> >> Thanks,
> >>
> >> Jun
> >>
> >>
> >> On Tue, Jul 23, 2013 at 7:58 AM, Anurup Raveendran <
> >> [EMAIL PROTECTED]> wrote:
> >>
> >> > After consuming about 5 gb of messages , it stopped consuming and it
> got
> >> > stuck at a particular offset. After running the following command I
> got
> >> an
> >> > error regarding the broker.
> >> >
> >> >
> >> >
> >> >
> >> > $ bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --group
> >> > air-DummyProductionConsumerGroup-After --zkconnect 172.16.18.65:2181
> >> --topic
> >> > air
> >> >
> >> >
> >> >
> >> > [2013-07-23 20:19:42,748] INFO Starting ZkClient event thread.
> >> > (org.I0Itec.zkclient.ZkEventThread)
> >> > [2013-07-23 20:19:42,757] INFO Client
> >> > environment:zookeeper.version=3.3.3-1203054, built on 11/17/2011 05:47
> >> GMT
> >> > (org.apache.zookeeper.ZooKeeper)
> >> > [2013-07-23 20:19:42,757] INFO Client environment:host.name=localhost
> >> > (org.apache.zookeeper.ZooKeeper)
> >> > [2013-07-23 20:19:42,757] INFO Client
> environment:java.version=1.7.0_21
> >> > (org.apache.zookeeper.ZooKeeper)
> >> > [2013-07-23 20:19:42,757] INFO Client environment:java.vendor=Oracle
> >> > Corporation (org.apache.zookeeper.ZooKeeper)
> >> > [2013-07-23 20:19:42,757] INFO Client
> >> > environment:java.home=/usr/java/jdk1.7.0_21/jre
> >> > (org.apache.zookeeper.ZooKeeper)
> >> > [2013-07-23 20:19:42,757] INFO Client
> >> >
> >> >
> >>
> environment:java.class.path=:bin/../project/boot/scala-2.8.0/lib/scala-compiler.jar:bin/../project/boot/scala-2.8.0/lib/scala-library.jar:bin/../core/target/scala_2.8.0/kafka-0.7.2.jar:bin/../core/lib/*.jar:bin/../perf/target/scala_2.8.0/kafka-perf-0.7.2.jar:bin/../core/lib_managed/scala_2.8.0/compile/jopt-simple-3.2.jar:bin/../core/lib_managed/scala_2.8.0/compile/log4j-1.2.15.jar:bin/../core/lib_managed/scala_2.8.0/compile/snappy-java-1.0.4.1.jar:bin/../core/lib_managed/scala_2.8.0/compile/zkclient-0.1.jar:bin/../core/lib_managed/scala_2.8.0/compile/zookeeper-3.3.4.jar
> >> > (org.apache.zookeeper.ZooKeeper)
> >> > [2013-07-23 20:19:42,758] INFO Client
> >> >
> >> >
> >>
> environment:java.library.path=/usr/local/oracle:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
> >> > (org.apache.zookeeper.ZooKeeper)
> >> > [2013-07-23 20:19:42,758] INFO Client environment:java.io.tmpdir=/tmp
> >> > (org.apache.zookeeper.ZooKeeper)
> >> > [2013-07-23 20:19:42,758] INFO Client environment:java.compiler=<NA>
> >> > (org.apache.zookeeper.ZooKeeper)
> >> > [2013-07-23 20:19:42,758] INFO Client environment:os.name=Linux
> >> > (org.apache.zookeeper.ZooKeeper)
> >> > [2013-07-23 20:19:42,758] INFO Client environment:os.arch=amd64
> >> > (org.apache.zookeeper.ZooKeeper)
> >> > [2013-07-23 20:19:42,758] INFO Client
> >> > environment:os.version=2.6.32-279.22.1.el6.x86_64

 
+
Samir Madhavan 2013-07-29, 10:06
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB