Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka, mail # user - consumer partition rebalancing


Copy link to this message
-
consumer partition rebalancing
Kam Kasravi 2013-09-10, 20:04
I wasreviewingtheconsumer partition rebalancingalgorithm and had a fewrelatedquestions 

* Assuming 1) theconsumerdoesn'tcontrolthe partition allocation within a topic and 2) theconstraintthat a single consumer C(i) within a consumer group C(g) must be the only reader of that partition:
* Are there ways to scale partition consumption if C(i) cannot keep up?
* A way to designate particular consumers should consume particular partitions based on consumer capability (similar to container allocation in YARN)?
* A way to designate sticky consumer / partition allocation so that a new consumer C(j) or new broker B(k) or new topic T(l) does not cause C(i) to lose particular partitions during rebalancing?

Thanks
Kam