Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka >> mail # user >> Apache Kafka in AWS


+
Jason Weiss 2013-05-22, 20:42
+
Neha Narkhede 2013-05-22, 20:57
+
Ken Krugler 2013-05-22, 21:24
+
Scott Clasen 2013-05-22, 23:27
+
Jonathan Hodges 2013-05-22, 23:11
+
Scott Clasen 2013-05-22, 23:56
+
Ken Krugler 2013-05-23, 01:00
+
Jun Rao 2013-05-23, 04:17
+
Jun Rao 2013-05-23, 14:11
+
Jason Weiss 2013-05-23, 14:13
Copy link to this message
-
Re: Apache Kafka in AWS
Curious if you tested with larger message sizes, like around 20-30kb (you
mentioned 2kb).

Any numbers on that size?
On Thu, May 23, 2013 at 10:12 AM, Jason Weiss <[EMAIL PROTECTED]>wrote:

> Bummer.
>
> Yes, but it will be several days. I'll post back to the forum with a URL
> once I'm done.
>
> Jason
>
>
>
> On 5/23/13 10:11 AM, "Jun Rao" <[EMAIL PROTECTED]> wrote:
>
> >Jason,
> >
> >Unfortunately, Apache mailing lists don't support attachments. Could you
> >document your experience (with the graphs) in a blog (or a wiki page in
> >Kafka)?
> >
> >Thanks,
> >
> >Jun
> >
> >
> >On Thu, May 23, 2013 at 2:00 AM, Jason Weiss <[EMAIL PROTECTED]>
> >wrote:
> >
> >> Jun,
> >>
> >> Here is a screenshot from AWS's statistics (per-minute sampling is the
> >> finest granularity I believe that they chart). I don't have a
> >>screenshot of
> >> the top output.
> >>
> >> This shows when I added a 4th machine to the cluster with the same
> >>number
> >> of clients, my CPU utilization fell- but remained constant. The
> >>flatline is
> >> pretty obvious in the extended 4 minute test-- it ramps up, flat lines,
> >> then ramps down.
> >>
> >> Jason
> >>
> >> ________________________________________
> >> From: Jun Rao [[EMAIL PROTECTED]]
> >> Sent: Thursday, May 23, 2013 00:17
> >> To: [EMAIL PROTECTED]
> >> Subject: Re: Apache Kafka in AWS
> >>
> >> Jason,
> >>
> >> Thanks for sharing. This is very interesting. Normally, Kafka brokers
> >>don't
> >> use too much CPU. Are most of the 750% CPU actually used by Kafka
> >>brokers?
> >>
> >> Jun
> >>
> >>
> >> On Wed, May 22, 2013 at 6:11 PM, Jason Weiss <[EMAIL PROTECTED]>
> >> wrote:
> >>
> >> > >>Did you check that you were using all cores?
> >> >
> >> > top was reporting over 750%
> >> >
> >> > Jason
> >> >
> >> > ________________________________________
> >> > From: Ken Krugler [[EMAIL PROTECTED]]
> >> > Sent: Wednesday, May 22, 2013 20:59
> >> > To: [EMAIL PROTECTED]
> >> > Subject: Re: Apache Kafka in AWS
> >> >
> >> > Hi Jason,
> >> >
> >> > On May 22, 2013, at 3:35pm, Jason Weiss wrote:
> >> >
> >> > > Ken,
> >> > >
> >> > > Great question! I should have indicated I was using EBS, 500GB with
> >> 2000
> >> > provisioned IOPs.
> >> >
> >> > OK, thanks. Sounds like you were pegged on CPU usage.
> >> >
> >> > But that does surprise me a bit. Did you check that you were using all
> >> > cores?
> >> >
> >> > Thanks,
> >> >
> >> > -- Ken
> >> >
> >> > PS - back in 2006 I spent a week of hell debugging an occasion job
> >> failure
> >> > on Hadoop (this is when it was still part of Nutch). Turns out one of
> >>our
> >> > 12 slaves was accidentally using OpenJDK, and this had a JIT compiler
> >>bug
> >> > that would occasionally rear its ugly head. Obviously the Sun/Oracle
> >>JRE
> >> > isn't bug-free, but it gets a lot more stress testing. So one of my
> >>basic
> >> > guidelines in the ops portion of the Hadoop class I teach is that
> >>every
> >> > server must have exactly the same version of Oracle's JRE.
> >> >
> >> > > ________________________________________
> >> > > From: Ken Krugler [[EMAIL PROTECTED]]
> >> > > Sent: Wednesday, May 22, 2013 17:23
> >> > > To: [EMAIL PROTECTED]
> >> > > Subject: Re: Apache Kafka in AWS
> >> > >
> >> > > Hi Jason,
> >> > >
> >> > > Thanks for the notes.
> >> > >
> >> > > I'm curious whether you went with using local drives (ephemeral
> >> storage)
> >> > or EBS, and if with EBS then what IOPS.
> >> > >
> >> > > Thanks,
> >> > >
> >> > > -- Ken
> >> > >
> >> > > On May 22, 2013, at 1:42pm, Jason Weiss wrote:
> >> > >
> >> > >> All,
> >> > >>
> >> > >> I asked a number of questions of the group over the last week, and
> >>I'm
> >> > happy to report that I've had great success getting Kafka up and
> >>running
> >> in
> >> > AWS. I am using 3 EC2 instances, each of which is a M2 High-Memory
> >> > Quadruple Extra Large with 8 cores and 58.4 GiB of memory according to
> >> the
> >> > AWS specs. I have co-located Zookeeper instances next to Zafka on each

 
+
S Ahmed 2013-05-29, 17:40