Kafka, mail # user - consumer partition rebalancing - 2013-09-10, 20:04
Solr & Elasticsearch trainings in New York & San Francisco [more info][hide]
 Search Hadoop and all its subprojects:

Switch to Threaded View
Copy link to this message
-
consumer partition rebalancing
I wasreviewingtheconsumer partition rebalancingalgorithm and had a fewrelatedquestions 

* Assuming 1) theconsumerdoesn'tcontrolthe partition allocation within a topic and 2) theconstraintthat a single consumer C(i) within a consumer group C(g) must be the only reader of that partition:
* Are there ways to scale partition consumption if C(i) cannot keep up?
* A way to designate particular consumers should consume particular partitions based on consumer capability (similar to container allocation in YARN)?
* A way to designate sticky consumer / partition allocation so that a new consumer C(j) or new broker B(k) or new topic T(l) does not cause C(i) to lose particular partitions during rebalancing?

Thanks
Kam
 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB