Home | About | Sematext search-lucene.com search-hadoop.com search-devops.com metrics + logs = try SPM and Logsene for free
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka >> mail # user >> Consumer rebalance per topic


+
Pablo Barrera González 2013-01-07, 17:04
Copy link to this message
-
Re: Consumer rebalance per topic
Pablo,

Currently, partition is the smallest unit that we distribute data among
consumers (in the same consumer group). So, if the # of consumers is larger
than the total number of partitions in a Kafka cluster (across all
brokers), some consumers will never get any data. Such a decision is done
on a per topic basis. If a consumer consumes multiple topics, it would make
sense to divide partitions across all topics to consumers. We haven't done
that yet. Part of the reason is that we need to figure out how to balance
the data across topics since they can be of different sizes. We can look
into that post 0.8.

For now, the solution is to increase the number of partitions on the broker.

Thanks,

Jun

On Mon, Jan 7, 2013 at 9:03 AM, Pablo Barrera González <
[EMAIL PROTECTED]> wrote:
 
+
Neha Narkhede 2013-01-07, 18:44
+
Pablo Barrera González 2013-01-07, 19:11
+
Pablo Barrera González 2013-01-08, 09:29
+
Joel Koshy 2013-01-08, 20:08
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB