Kafka, mail # user - Problem with consumers in a consumers group - 2013-05-09, 15:51
 Search Hadoop and all its subprojects:

Switch to Plain View
Copy link to this message
-
Problem with consumers in a consumers group
Hi.
I`m sending nginx logs to kafka. Each nginx goes to one exclusive partition.
I guarantee this by asigning to each nginx a number (1 to 22), and I use
that number for a custom partitoner's partition method as the key parameter:
  return Integer.parseInt(key) % partitions

I have 22 nginxs, so I have 22 partitions with data.
The cluster has 5 brokers with ten partitions, so the total partitions for
the topic is 50.

In teory I could be able to start up to 22 consumers and each one should be
able to consume from one partition.
But what is happening is that I only can start three consumers:
The first one consumes the ten partitions of the first broker.
The second one consumes the ten partitions of the second broker.
The third one consumes the remaining 2 partitions of the third broker.

Any other consumer that I start in the same consumer group, doesn`t
consumes anything, and the three consumers that are consuming are not
balanced.

I am using Kafka 0.7.1, with zookeeper and the java consumer connector,
with the zk.connect property.

What could it be happening?

Thanks for the help!

Javier Iseruk
Architecture Team
Phone: +5411-4640-8063

 
+
Neha Narkhede 2013-05-09, 16:22
+
Javier Iseruk 2013-05-09, 18:41
+
Javier Iseruk 2013-05-09, 20:18
+
Dennis Haller 2013-05-13, 15:09
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB