Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka >> mail # user >> Problem with consumers in a consumers group


+
Javier Iseruk 2013-05-09, 15:51
+
Neha Narkhede 2013-05-09, 16:22
+
Javier Iseruk 2013-05-09, 18:41
Copy link to this message
-
Re: Problem with consumers in a consumers group
Hi again!
I have more info to share.
What I'm seeing in zookeeper is that the consumers are owning almost the
same partitions quantity, no matter if that partition has data or not. So
for example:
Consumer 1: 17 partitions (all with data)
Consumer 2: 17 partitions (only five with data)
Consumer 3: 16 partitions (no data at all)

That is is the photo I'm seeing now.

I would expect that the consumers own the same not empty partitions
quantity. I don`t care about the empty partitions, but apparently, Kafka
divides the partitions no matter if it`s an empty partition or not.
So I would like to see something like this:
Consumer 1: 17 partitions (7 with data)
Consumer 2: 17 partitions (7 with data)
Consumer 3: 16 partitions (8 data at all)

Or if I add another consumer, the result expected would be something like
this:
Consumer 1: 13 partitions (5 with data)
Consumer 2: 13 partitions (5 with data)
Consumer 3: 12 partitions (6 data at all)
Consumer 4: 12 partitions (6 data at all)

Am I right or wrong?

Thanks for the help!!! (And sorry for my english)

On Thu, May 9, 2013 at 3:40 PM, Javier Iseruk <
[EMAIL PROTECTED]> wrote:

> Thanks for the answer. I send you the info:
> I am ommiting all the files in each directory, just leaves one for showing
> the structure.
> There are two topics: access_log@nginx-nginx-api-master and
> access_log@nginx-nginx-api-v1.
> The consumer uses a whitelist with the regex access_log@nginx-nginx-api-.*,
> so both topis are consumed
>  I don't see any problem with the topics names.
>
> Broker 0:
> /kafka/logs> tree
> .
> |-- access_log@nginx-nginx-api-master-0
> |   `-- 00000000104220676901.kafka
> |-- access_log@nginx-nginx-api-master-1
> |   `-- 00000001848656795194.kafka
> |-- access_log@nginx-nginx-api-master-2
> |   `-- 00000000205422337816.kafka
> |-- access_log@nginx-nginx-api-master-3
> |   `-- 00000000065714271675.kafka
> |-- access_log@nginx-nginx-api-master-4
> |   `-- 00000002042994150834.kafka
> |-- access_log@nginx-nginx-api-master-5
> |   `-- 00000003554704806448.kafka
> |-- access_log@nginx-nginx-api-master-6
> |   `-- 00000005445803344687.kafka
> |-- access_log@nginx-nginx-api-master-7
> |   `-- 00000003606342657110.kafka
> |-- access_log@nginx-nginx-api-master-8
> |   `-- 00000002693751681000.kafka
> |-- access_log@nginx-nginx-api-master-9
> |   `-- 00000001393405947605.kafka
> |-- access_log@nginx-nginx-api-v1-0
> |   `-- 00000000132598986468.kafka
> |-- access_log@nginx-nginx-api-v1-1
> |   `-- 00000000137269970386.kafka
> |-- access_log@nginx-nginx-api-v1-2
> |   `-- 00000000136972308161.kafka
> |-- access_log@nginx-nginx-api-v1-3
> |   `-- 00000000146486674802.kafka
> |-- access_log@nginx-nginx-api-v1-6
> |   `-- 00000000009125337012.kafka
> |-- access_log@nginx-nginx-api-v1-8
> |   `-- 00000000009010149647.kafka
>
> broker 1:
> /kafka/logs> tree
> .
> |-- access_log@nginx-nginx-api-master-0
> |   `-- 00000002313153902757.kafka
> |-- access_log@nginx-nginx-api-master-1
> |   `-- 00000003097259627500.kafka
> |-- access_log@nginx-nginx-api-master-2
> |   `-- 00000002959536779137.kafka
> |-- access_log@nginx-nginx-api-master-3
> |   `-- 00000003053863102518.kafka
> |-- access_log@nginx-nginx-api-master-4
> |   `-- 00000002187535473763.kafka
> |-- access_log@nginx-nginx-api-master-5
> |   `-- 00000004427181341253.kafka
> |-- access_log@nginx-nginx-api-master-6
> |   `-- 00000002356845031168.kafka
> |-- access_log@nginx-nginx-api-master-7
> |   `-- 00000002286128315593.kafka
> |-- access_log@nginx-nginx-api-master-8
> |   `-- 00000002240322631141.kafka
> |-- access_log@nginx-nginx-api-master-9
> |   `-- 00000002298179990598.kafka
> |-- access_log@nginx-nginx-api-v1-0
> |   `-- 00000000000111649340.kafka
> |-- access_log@nginx-nginx-api-v1-1
> |   `-- 00000000000112654964.kafka
> |-- access_log@nginx-nginx-api-v1-2
> |   `-- 00000000000110500958.kafka
> |-- access_log@nginx-nginx-api-v1-3
> |   `-- 00000000000107970659.kafka
>
> broker 2:
> /kafka/logs> tree
> .
> |-- access_log@nginx-nginx-api-master-0
Javier Iseruk
Architecture Team
Phone: +5411-4640-8063

 
+
Dennis Haller 2013-05-13, 15:09
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB