Kafka, mail # user - Kafka consumer not consuming events - 2013-07-09, 11:19
Solr & Elasticsearch trainings in New York & San Francisco [more info][hide]
 Search Hadoop and all its subprojects:

Switch to Plain View
Jun Rao 2013-07-22, 05:22
Copy link to this message
Kafka consumer not consuming events

We are using kafka-0.7.2 with zookeeper (3.4.5)

Our cluster configuration:
3 brokers on 3 different machines. Each broker machine has a zookeeper instance running as well.
We have 15 topics defined. We are trying to use them as queue (JMS like) by defining the same group across different kafka consumers.
On the consumer side, we are using High Level Consumer.

However we are seeing a weird behaviour.
One of our heavily used queue (event_queue) has 2 dedicated consumers listening to that queue only.
This queue is defined with 150 partitions on each broker & the number of streams defined on the 2 dedicated consumers is 150.
After a while we see that most the consumer threads keep waiting for events and the lag keeps growing.
If we kill one of the dedicated consumers, then the other consumer starts getting messaging in a hurry.

Consumer had no Full GCs.

How we measure lag?
bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --group event_queue --zkconnect zookeeper1:2181,zookeeper2:2181,zookeeper3:2181/kafka --topic event_queue

Around the time, the events stopped coming to the new consumer.. this was printed on the logs:

[INFO] zookeeper state changed (Disconnected)
[INFO] zookeeper state changed (Disconnected)
[INFO] zookeeper state changed (SyncConnected)
[INFO] zookeeper state changed (SyncConnected)

Config Overidden:

Please let us know if we are doing some wrong OR facing some known issue here?

Jun Rao 2013-07-09, 15:03
Nihit Purwar 2013-07-09, 15:38
Jun Rao 2013-07-10, 03:43
Nihit Purwar 2013-07-10, 06:12
Jun Rao 2013-07-10, 15:00
Nihit Purwar 2013-07-10, 15:31
Jun Rao 2013-07-10, 16:18
Jun Rao 2013-07-10, 17:09
Nihit Purwar 2013-07-11, 11:12
Nihit Purwar 2013-07-19, 07:31
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB