Re: Random Partitioning Issue
Thanks for bringing this up. I want to clarify this a bit.
1. Currently, the producer side logic is that if the partitioning key is
not provided (i.e., it is null), the partitioner won't be called. We did
that because we want to select a random and "available" partition to send
messages so that if some partitions are temporarily unavailable (because of
broker failures), messages can still be sent to other partitions. Doing
this in the partitioner is difficult since the partitioner doesn't know
which partitions are currently available (the DefaultEventHandler does).
2. As Joel said, the common use case in production is that there are many
more producers than #partitions in a topic. In this case, sticking to a
partition for a few minutes is not going to cause too much imbalance in the
partitions and has the benefit of reducing the # of socket connections. My
feeling is that this will benefit most production users. In fact, if one
uses a hardware load balancer for producing data in 0.7, it behaves in
exactly the same way (a producer will stick to a broker until the reconnect
interval is reached).
3. It is true that If one is testing a topic with more than one partition
(which is not the default value), this behavior can be a bit weird.
However, I think it can be mitigated by running multiple test producer
4. Someone reported in the mailing list that all data shows in only one
partition after a few weeks. This is clearly not the expected behavior. We
can take a closer look to see if this is real issue.
Do you think these address your concerns?
On Sat, Sep 14, 2013 at 11:18 AM, Joe Stein <[EMAIL PROTECTED]> wrote: