Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka, mail # user - High-level consumer load-balancing problem


Copy link to this message
-
Re: High-level consumer load-balancing problem
Jun Rao 2013-11-15, 05:03
The consumer load balancing logic today is pretty simple. It just tries to
divide the partitions evenly among the consumers. It doesn't try to balance
by load.

Thanks,

Jun
On Thu, Nov 14, 2013 at 3:34 PM, [EMAIL PROTECTED] <[EMAIL PROTECTED]> wrote:

> Hi,
>
> I have questions about the load balancing of kafka high-level consumer
> Suppose I have 4 partition
> And the producer throughput to these 4 partitions are like this
> 0                    1                   2                3
> 10MB/s          10MB/s          1MB/s        1MB/s
> 1kMsg/s,        10kMsg/s       1kMsg/s     10kMsg/s
>
> And I have only 2 consumers, what partitions will the 2 consumers pick up?
>
> If the throughput is changing dynamically, will the consumer re-balancing
> accordingly?
>
> Thanks,
> Siyuan
>