Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka >> mail # user >> Kafka consumer not consuming events


+
Nihit Purwar 2013-07-09, 11:19
+
Jun Rao 2013-07-09, 15:03
+
Nihit Purwar 2013-07-09, 15:38
Copy link to this message
-
Re: Kafka consumer not consuming events
This indicates our in-memory queue is empty. So the consumer thread is
blocked. What about the Kafka fetcher threads? Are they blocked on anything?

Thanks,

Jun
On Tue, Jul 9, 2013 at 8:37 AM, Nihit Purwar <[EMAIL PROTECTED]> wrote:

> Hello Jun,
>
> Please see my comments inline.
>
> On 09-Jul-2013, at 8:32 PM, Jun Rao <[EMAIL PROTECTED]> wrote:
>
> > I assume that each consumer instance consumes all 15 topics.
> No, we kept dedicated consumer listening to the topic in question.
> We did this because this queue processes huge amounts of data.
>
>
> > Are all your
> > consumer threads alive? If one of your thread dies, it will eventually
> > block the consumption in other threads.
>
> Yes. We can see all the threads in the thread dump.
> We have ensured that the threads do not die due to an Exception.
>
> Please look at the stack trace below. We see all the threads waiting like
> this:
>
> "event_queue@150" prio=10 tid=0x00007eff28e41800 nid=0x31f9 waiting on
> condition [0x00007efedae6d000]
>    java.lang.Thread.State: WAITING (parking)
>         at sun.misc.Unsafe.park(Native Method)
>         - parking to wait for  <0x0000000640248618> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>         at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
>         at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
>         at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
>         at
> kafka.consumer.ConsumerIterator.makeNext(ConsumerIterator.scala:60)
>         at
> kafka.consumer.ConsumerIterator.makeNext(ConsumerIterator.scala:32)
>         at
> kafka.utils.IteratorTemplate.maybeComputeNext(IteratorTemplate.scala:59)
>         at kafka.utils.IteratorTemplate.hasNext(IteratorTemplate.scala:51)
>         at
> com.spr.messageprocessor.KafkaStreamRunnable.run(KafkaStreamRunnable.java:49)
>         at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
>         at
> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>         at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>         at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>         at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>         at java.lang.Thread.run(Thread.java:662)
>
> >
> > Thanks,
> >
> > Jun
> >
> >
> > On Tue, Jul 9, 2013 at 4:18 AM, Nihit Purwar <[EMAIL PROTECTED]>
> wrote:
> >
> >> Hi,
> >>
> >> We are using kafka-0.7.2 with zookeeper (3.4.5)
> >>
> >> Our cluster configuration:
> >> 3 brokers on 3 different machines. Each broker machine has a zookeeper
> >> instance running as well.
> >> We have 15 topics defined. We are trying to use them as queue (JMS like)
> >> by defining the same group across different kafka consumers.
> >> On the consumer side, we are using High Level Consumer.
> >>
> >> However we are seeing a weird behaviour.
> >> One of our heavily used queue (event_queue) has 2 dedicated consumers
> >> listening to that queue only.
> >> This queue is defined with 150 partitions on each broker & the number of
> >> streams defined on the 2 dedicated consumers is 150.
> >> After a while we see that most the consumer threads keep waiting for
> >> events and the lag keeps growing.
> >> If we kill one of the dedicated consumers, then the other consumer
> starts
> >> getting messaging in a hurry.
> >>
> >> Consumer had no Full GCs.
> >>
> >> How we measure lag?
> >> bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --group
> >> event_queue --zkconnect
> >> zookeeper1:2181,zookeeper2:2181,zookeeper3:2181/kafka --topic
> event_queue
> >>
> >> Around the time, the events stopped coming to the new consumer.. this
> was
> >> printed on the logs:
> >>
> >> [INFO] zookeeper state changed (Disconnected)
> >> [INFO] zookeeper state changed (Disconnected)
> >> [INFO] zookeeper state changed (SyncConnected)

 
+
Nihit Purwar 2013-07-10, 06:12
+
Jun Rao 2013-07-10, 15:00
+
Nihit Purwar 2013-07-10, 15:31
+
Jun Rao 2013-07-10, 16:18
+
Jun Rao 2013-07-10, 17:09
+
Nihit Purwar 2013-07-11, 11:12
+
Nihit Purwar 2013-07-19, 07:31
+
Jun Rao 2013-07-22, 05:22
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB