Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Apache Kafka in AWS


Copy link to this message
-
Re: Apache Kafka in AWS
Is the code you used to benchmark open source by any chance?
On Tue, May 28, 2013 at 4:29 PM, Jason Weiss <[EMAIL PROTECTED]> wrote:

> Nope, sorry.
>
>
> ________________________________________
> From: S Ahmed [[EMAIL PROTECTED]]
> Sent: Tuesday, May 28, 2013 15:47
> To: [EMAIL PROTECTED]
> Subject: Re: Apache Kafka in AWS
>
> Curious if you tested with larger message sizes, like around 20-30kb (you
> mentioned 2kb).
>
> Any numbers on that size?
>
>
> On Thu, May 23, 2013 at 10:12 AM, Jason Weiss <[EMAIL PROTECTED]
> >wrote:
>
> > Bummer.
> >
> > Yes, but it will be several days. I'll post back to the forum with a URL
> > once I'm done.
> >
> > Jason
> >
> >
> >
> > On 5/23/13 10:11 AM, "Jun Rao" <[EMAIL PROTECTED]> wrote:
> >
> > >Jason,
> > >
> > >Unfortunately, Apache mailing lists don't support attachments. Could you
> > >document your experience (with the graphs) in a blog (or a wiki page in
> > >Kafka)?
> > >
> > >Thanks,
> > >
> > >Jun
> > >
> > >
> > >On Thu, May 23, 2013 at 2:00 AM, Jason Weiss <[EMAIL PROTECTED]>
> > >wrote:
> > >
> > >> Jun,
> > >>
> > >> Here is a screenshot from AWS's statistics (per-minute sampling is the
> > >> finest granularity I believe that they chart). I don't have a
> > >>screenshot of
> > >> the top output.
> > >>
> > >> This shows when I added a 4th machine to the cluster with the same
> > >>number
> > >> of clients, my CPU utilization fell- but remained constant. The
> > >>flatline is
> > >> pretty obvious in the extended 4 minute test-- it ramps up, flat
> lines,
> > >> then ramps down.
> > >>
> > >> Jason
> > >>
> > >> ________________________________________
> > >> From: Jun Rao [[EMAIL PROTECTED]]
> > >> Sent: Thursday, May 23, 2013 00:17
> > >> To: [EMAIL PROTECTED]
> > >> Subject: Re: Apache Kafka in AWS
> > >>
> > >> Jason,
> > >>
> > >> Thanks for sharing. This is very interesting. Normally, Kafka brokers
> > >>don't
> > >> use too much CPU. Are most of the 750% CPU actually used by Kafka
> > >>brokers?
> > >>
> > >> Jun
> > >>
> > >>
> > >> On Wed, May 22, 2013 at 6:11 PM, Jason Weiss <[EMAIL PROTECTED]>
> > >> wrote:
> > >>
> > >> > >>Did you check that you were using all cores?
> > >> >
> > >> > top was reporting over 750%
> > >> >
> > >> > Jason
> > >> >
> > >> > ________________________________________
> > >> > From: Ken Krugler [[EMAIL PROTECTED]]
> > >> > Sent: Wednesday, May 22, 2013 20:59
> > >> > To: [EMAIL PROTECTED]
> > >> > Subject: Re: Apache Kafka in AWS
> > >> >
> > >> > Hi Jason,
> > >> >
> > >> > On May 22, 2013, at 3:35pm, Jason Weiss wrote:
> > >> >
> > >> > > Ken,
> > >> > >
> > >> > > Great question! I should have indicated I was using EBS, 500GB
> with
> > >> 2000
> > >> > provisioned IOPs.
> > >> >
> > >> > OK, thanks. Sounds like you were pegged on CPU usage.
> > >> >
> > >> > But that does surprise me a bit. Did you check that you were using
> all
> > >> > cores?
> > >> >
> > >> > Thanks,
> > >> >
> > >> > -- Ken
> > >> >
> > >> > PS - back in 2006 I spent a week of hell debugging an occasion job
> > >> failure
> > >> > on Hadoop (this is when it was still part of Nutch). Turns out one
> of
> > >>our
> > >> > 12 slaves was accidentally using OpenJDK, and this had a JIT
> compiler
> > >>bug
> > >> > that would occasionally rear its ugly head. Obviously the Sun/Oracle
> > >>JRE
> > >> > isn't bug-free, but it gets a lot more stress testing. So one of my
> > >>basic
> > >> > guidelines in the ops portion of the Hadoop class I teach is that
> > >>every
> > >> > server must have exactly the same version of Oracle's JRE.
> > >> >
> > >> > > ________________________________________
> > >> > > From: Ken Krugler [[EMAIL PROTECTED]]
> > >> > > Sent: Wednesday, May 22, 2013 17:23
> > >> > > To: [EMAIL PROTECTED]
> > >> > > Subject: Re: Apache Kafka in AWS
> > >> > >
> > >> > > Hi Jason,
> > >> > >
> > >> > > Thanks for the notes.
> > >> > >
> > >> > > I'm curious whether you went with using local drives (ephemeral

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB