Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka >> mail # user >> Problem with consumers in a consumers group


+
Javier Iseruk 2013-05-09, 15:51
Copy link to this message
-
Re: Problem with consumers in a consumers group
Could you check if the topic name is the same on all 5 brokers ? Can you
send around the contents of the kafka log directory on all 5 brokers ?

Thanks,
Neha
On Thu, May 9, 2013 at 8:51 AM, Javier Iseruk <
[EMAIL PROTECTED]> wrote:

> Hi.
> I`m sending nginx logs to kafka. Each nginx goes to one exclusive
> partition.
> I guarantee this by asigning to each nginx a number (1 to 22), and I use
> that number for a custom partitoner's partition method as the key
> parameter:
>   return Integer.parseInt(key) % partitions
>
> I have 22 nginxs, so I have 22 partitions with data.
> The cluster has 5 brokers with ten partitions, so the total partitions for
> the topic is 50.
>
> In teory I could be able to start up to 22 consumers and each one should be
> able to consume from one partition.
> But what is happening is that I only can start three consumers:
> The first one consumes the ten partitions of the first broker.
> The second one consumes the ten partitions of the second broker.
> The third one consumes the remaining 2 partitions of the third broker.
>
> Any other consumer that I start in the same consumer group, doesn`t
> consumes anything, and the three consumers that are consuming are not
> balanced.
>
> I am using Kafka 0.7.1, with zookeeper and the java consumer connector,
> with the zk.connect property.
>
> What could it be happening?
>
> Thanks for the help!
>
> --
> Javier Iseruk
> Architecture Team
> Phone: +5411-4640-8063
>

 
+
Javier Iseruk 2013-05-09, 18:41
+
Javier Iseruk 2013-05-09, 20:18
+
Dennis Haller 2013-05-13, 15:09
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB