Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Best partition configuration


Copy link to this message
-
Re: Best partition configuration
Hey Tom,

I tried to expand the FAQ on this, let me know if you feel like it answers
your question:

https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-HowdoIchoosethenumberofpartitionsforatopic%3F

Folks--let me know if I missed anything there. I tried to give both a
simple answer and a more complete answer.

-Jay

On Wed, Aug 21, 2013 at 9:52 AM, Tom Brown <[EMAIL PROTECTED]> wrote:

> Philip,
>
> How many topics per broker (just one?) And what is the read/write profile
> of your setup?
>
> --Tom
>
>
> On Wed, Aug 21, 2013 at 12:24 PM, Philip O'Toole <[EMAIL PROTECTED]>
> wrote:
>
> > On Wed, Aug 21, 2013 at 10:06:50AM -0400, Tom Brown wrote:
> > > I have not been able to find reliable advice regarding how many
> > partitions
> > > should exist on a single broker. How many partitions have you used, and
> > > what kind of throughput have you seen?
> >
> > We settled on 8 per broker, and usually run 3 brokers in production
> > environments, giving us a total of 24 partitions. Throughput has been
> > superb.
> >
> > For integration testing however, we usually use just 1 or 2 partitions.
> >
> > Philip
> >
> > >
> > > Thanks in advance!
> > >
> > > --Tom
> >
> > --
> > Philip O'Toole
> >
> > Senior Developer
> > Loggly, Inc.
> > San Francisco, CA.
> > www.loggly.com
> >
> > Come join us!
> > http://loggly.com/company/careers/
> >
>

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB