Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Apache Kafka in AWS


Copy link to this message
-
Re: Apache Kafka in AWS
Is the code you used to benchmark open source by any chance?
On Tue, May 28, 2013 at 4:29 PM, Jason Weiss <[EMAIL PROTECTED]> wrote:

> Nope, sorry.
>
>
> ________________________________________
> From: S Ahmed [[EMAIL PROTECTED]]
> Sent: Tuesday, May 28, 2013 15:47
> To: [EMAIL PROTECTED]
> Subject: Re: Apache Kafka in AWS
>
> Curious if you tested with larger message sizes, like around 20-30kb (you
> mentioned 2kb).
>
> Any numbers on that size?
>
>
> On Thu, May 23, 2013 at 10:12 AM, Jason Weiss <[EMAIL PROTECTED]
> >wrote:
>
> > Bummer.
> >
> > Yes, but it will be several days. I'll post back to the forum with a URL
> > once I'm done.
> >
> > Jason
> >
> >
> >
> > On 5/23/13 10:11 AM, "Jun Rao" <[EMAIL PROTECTED]> wrote:
> >
> > >Jason,
> > >
> > >Unfortunately, Apache mailing lists don't support attachments. Could you
> > >document your experience (with the graphs) in a blog (or a wiki page in
> > >Kafka)?
> > >
> > >Thanks,
> > >
> > >Jun
> > >
> > >
> > >On Thu, May 23, 2013 at 2:00 AM, Jason Weiss <[EMAIL PROTECTED]>
> > >wrote:
> > >
> > >> Jun,
> > >>
> > >> Here is a screenshot from AWS's statistics (per-minute sampling is the
> > >> finest granularity I believe that they chart). I don't have a
> > >>screenshot of
> > >> the top output.
> > >>
> > >> This shows when I added a 4th machine to the cluster with the same
> > >>number
> > >> of clients, my CPU utilization fell- but remained constant. The
> > >>flatline is
> > >> pretty obvious in the extended 4 minute test-- it ramps up, flat
> lines,
> > >> then ramps down.
> > >>
> > >> Jason
> > >>
> > >> ________________________________________
> > >> From: Jun Rao [[EMAIL PROTECTED]]
> > >> Sent: Thursday, May 23, 2013 00:17
> > >> To: [EMAIL PROTECTED]
> > >> Subject: Re: Apache Kafka in AWS
> > >>
> > >> Jason,
> > >>
> > >> Thanks for sharing. This is very interesting. Normally, Kafka brokers
> > >>don't
> > >> use too much CPU. Are most of the 750% CPU actually used by Kafka
> > >>brokers?
> > >>
> > >> Jun
> > >>
> > >>
> > >> On Wed, May 22, 2013 at 6:11 PM, Jason Weiss <[EMAIL PROTECTED]>
> > >> wrote:
> > >>
> > >> > >>Did you check that you were using all cores?
> > >> >
> > >> > top was reporting over 750%
> > >> >
> > >> > Jason
> > >> >
> > >> > ________________________________________
> > >> > From: Ken Krugler [[EMAIL PROTECTED]]
> > >> > Sent: Wednesday, May 22, 2013 20:59
> > >> > To: [EMAIL PROTECTED]
> > >> > Subject: Re: Apache Kafka in AWS
> > >> >
> > >> > Hi Jason,
> > >> >
> > >> > On May 22, 2013, at 3:35pm, Jason Weiss wrote:
> > >> >
> > >> > > Ken,
> > >> > >
> > >> > > Great question! I should have indicated I was using EBS, 500GB
> with
> > >> 2000
> > >> > provisioned IOPs.
> > >> >
> > >> > OK, thanks. Sounds like you were pegged on CPU usage.
> > >> >
> > >> > But that does surprise me a bit. Did you check that you were using
> all
> > >> > cores?
> > >> >
> > >> > Thanks,
> > >> >
> > >> > -- Ken
> > >> >
> > >> > PS - back in 2006 I spent a week of hell debugging an occasion job
> > >> failure
> > >> > on Hadoop (this is when it was still part of Nutch). Turns out one
> of
> > >>our
> > >> > 12 slaves was accidentally using OpenJDK, and this had a JIT
> compiler
> > >>bug
> > >> > that would occasionally rear its ugly head. Obviously the Sun/Oracle
> > >>JRE
> > >> > isn't bug-free, but it gets a lot more stress testing. So one of my
> > >>basic
> > >> > guidelines in the ops portion of the Hadoop class I teach is that
> > >>every
> > >> > server must have exactly the same version of Oracle's JRE.
> > >> >
> > >> > > ________________________________________
> > >> > > From: Ken Krugler [[EMAIL PROTECTED]]
> > >> > > Sent: Wednesday, May 22, 2013 17:23
> > >> > > To: [EMAIL PROTECTED]
> > >> > > Subject: Re: Apache Kafka in AWS
> > >> > >
> > >> > > Hi Jason,
> > >> > >
> > >> > > Thanks for the notes.
> > >> > >
> > >> > > I'm curious whether you went with using local drives (ephemeral