Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Best partition configuration


Copy link to this message
-
Re: Best partition configuration
On Wed, Aug 21, 2013 at 10:06:50AM -0400, Tom Brown wrote:
> I have not been able to find reliable advice regarding how many partitions
> should exist on a single broker. How many partitions have you used, and
> what kind of throughput have you seen?

We settled on 8 per broker, and usually run 3 brokers in production
environments, giving us a total of 24 partitions. Throughput has been superb.

For integration testing however, we usually use just 1 or 2 partitions.

Philip

>
> Thanks in advance!
>
> --Tom

--
Philip O'Toole

Senior Developer
Loggly, Inc.
San Francisco, CA.
www.loggly.com

Come join us!
http://loggly.com/company/careers/

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB