Home | About | Sematext search-lucene.com search-hadoop.com search-devops.com metrics + logs = try SPM and Logsene for free
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # dev >> Random Partitioning Issue


Copy link to this message
-
Re: Random Partitioning Issue
How about creating a new class called RandomRefreshPartioner and copy the DefaultPartitioner code to it and then revert the DefaultPartitioner code.  I appreciate this is a one time burden for folks using the existing 0.8-beta1 bumping into KAFKA-1017 in production having to switch to the RandomRefreshPartioner and when folks deploy to production will have to consider this property change.

I make this suggestion keeping in mind the new folks that on board with Kafka and when everyone is in development and testing mode for the first time their experience would be as expected from how it would work in production this way.  In dev/test when first using Kafka they won't have so many producers for partitions but would look to parallelize their consumers IMHO.

The random broker change sounds like maybe a bigger change now this late in the release cycle if we can accommodate folks trying Kafka for the first time and through their development and testing along with full blown production deploys.

/*******************************************
 Joe Stein
 Founder, Principal Consultant
 Big Data Open Source Security LLC
 http://www.stealth.ly
 Twitter: @allthingshadoop
********************************************/
On Sep 14, 2013, at 8:17 AM, Joel Koshy <[EMAIL PROTECTED]> wrote:
 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB