Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka, mail # user - Re: partitioning


Copy link to this message
-
Re: partitioning
Stan Rosenberg 2013-01-14, 16:05
On Fri, Jan 11, 2013 at 12:37 AM, Jun Rao <[EMAIL PROTECTED]> wrote:

> Our current partitioning strategy is to mod key by # of partitions, not #
> brokers. For better balancing partitions over brokers, one simple strategy
> is to over partition, i.e., you have a few times of more partitions than
> brokers. That way, if one adds more brokers overtime, you can just move
> some existing partitions to the new broker.
>
> Consistent hashing requires changing # partitions dynamically.However, for
> some applications, they may prefer not to change partitions.
>

> What's your use case for consistent hashing?
>

My use case is essentially the same as above, i.e., dynamic load balancing.
 I now understand why the current partitioning strategy is used as opposed
to consistent hashing; partition "stickiness"
is definitely to be desired for the sake of moving computation to data.
However, the dynamic rebalancing as described in "Kafka Replication
Design", sect. 1.2 looks very similar to what's typically achieved by using
consistent hashing.
Is this rebalancing implemented in 0.8 or am I reading the now obsolete
documentation? :)  (If yes, could you please refer me to the code.)

Thanks,

stan