Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka, mail # user - Best partition configuration


+
Tom Brown 2013-08-21, 14:07
+
Philip OToole 2013-08-21, 16:24
+
Tom Brown 2013-08-21, 16:52
Copy link to this message
-
Re: Best partition configuration
Jay Kreps 2013-08-21, 18:12
Hey Tom,

I tried to expand the FAQ on this, let me know if you feel like it answers
your question:

https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-HowdoIchoosethenumberofpartitionsforatopic%3F

Folks--let me know if I missed anything there. I tried to give both a
simple answer and a more complete answer.

-Jay

On Wed, Aug 21, 2013 at 9:52 AM, Tom Brown <[EMAIL PROTECTED]> wrote:

> Philip,
>
> How many topics per broker (just one?) And what is the read/write profile
> of your setup?
>
> --Tom
>
>
> On Wed, Aug 21, 2013 at 12:24 PM, Philip O'Toole <[EMAIL PROTECTED]>
> wrote:
>
> > On Wed, Aug 21, 2013 at 10:06:50AM -0400, Tom Brown wrote:
> > > I have not been able to find reliable advice regarding how many
> > partitions
> > > should exist on a single broker. How many partitions have you used, and
> > > what kind of throughput have you seen?
> >
> > We settled on 8 per broker, and usually run 3 brokers in production
> > environments, giving us a total of 24 partitions. Throughput has been
> > superb.
> >
> > For integration testing however, we usually use just 1 or 2 partitions.
> >
> > Philip
> >
> > >
> > > Thanks in advance!
> > >
> > > --Tom
> >
> > --
> > Philip O'Toole
> >
> > Senior Developer
> > Loggly, Inc.
> > San Francisco, CA.
> > www.loggly.com
> >
> > Come join us!
> > http://loggly.com/company/careers/
> >
>

 
+
Philip OToole 2013-08-21, 18:26