Kafka, mail # dev - Re: Random Partitioning Issue - 2013-09-27, 16:46
Solr & Elasticsearch trainings in New York & San Francisco [more info][hide]
 Search Hadoop and all its subprojects:

Switch to Threaded View
Copy link to this message
-
Re: Random Partitioning Issue
Joe,

Not sure I fully understand your propose. Do you want to put the random
partitioning selection logic (for messages without a key) in the
partitioner without changing the partitioner api? That's difficult. The
issue is that in the current partitioner api, we don't know which
partitions are available. For example, if we have replication factor 1 on a
topic and a broker is down, the best thing to do for the random partitioner
is to select an available partition at random (assuming more than 1
partition is created for the topic).

Another option is to revert the logic in the random partitioning selection
logic in DefaultEventHandler to select a random partition per batch of
events (instead of sticking with a random partition for some configured
amount of time). This is doable, but I am not sure if it's that critical.
Since this is one of the two possible behaviors in 0.7, it's hard to say
whether people will be surprised by that. Preserving both behaviors in 0.7
will require changing the partitioner api. This is more work and I agree
it's better to do this post 0.8.0 final.

Thanks,

Jun

On Fri, Sep 27, 2013 at 9:24 AM, Joe Stein <[EMAIL PROTECTED]> wrote:
 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB