Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka, mail # user - Problem with consumers in a consumers group


Copy link to this message
-
Re: Problem with consumers in a consumers group
Neha Narkhede 2013-05-09, 16:22
Could you check if the topic name is the same on all 5 brokers ? Can you
send around the contents of the kafka log directory on all 5 brokers ?

Thanks,
Neha
On Thu, May 9, 2013 at 8:51 AM, Javier Iseruk <
[EMAIL PROTECTED]> wrote:

> Hi.
> I`m sending nginx logs to kafka. Each nginx goes to one exclusive
> partition.
> I guarantee this by asigning to each nginx a number (1 to 22), and I use
> that number for a custom partitoner's partition method as the key
> parameter:
>   return Integer.parseInt(key) % partitions
>
> I have 22 nginxs, so I have 22 partitions with data.
> The cluster has 5 brokers with ten partitions, so the total partitions for
> the topic is 50.
>
> In teory I could be able to start up to 22 consumers and each one should be
> able to consume from one partition.
> But what is happening is that I only can start three consumers:
> The first one consumes the ten partitions of the first broker.
> The second one consumes the ten partitions of the second broker.
> The third one consumes the remaining 2 partitions of the third broker.
>
> Any other consumer that I start in the same consumer group, doesn`t
> consumes anything, and the three consumers that are consuming are not
> balanced.
>
> I am using Kafka 0.7.1, with zookeeper and the java consumer connector,
> with the zk.connect property.
>
> What could it be happening?
>
> Thanks for the help!
>
> --
> Javier Iseruk
> Architecture Team
> Phone: +5411-4640-8063
>