Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka, mail # user - since flushes are batched, is it still io intensive?


+
S Ahmed 2012-05-11, 14:13
+
Jun Rao 2012-05-11, 14:34
Copy link to this message
-
Re: since flushes are batched, is it still io intensive?
S Ahmed 2012-05-11, 15:03
What about memory?  I know you guys have 24GB of ram per server?

Basically I'm juggling between going with a dedicated box (which has faster
IO), or ec2 which has slower IO but cheaper on the ram side (way cheaper!).

On Fri, May 11, 2012 at 10:34 AM, Jun Rao <[EMAIL PROTECTED]> wrote:

> It all depends on the volume of the data. At LinkedIn, we observed that the
> io load on a typical Kafka broker is not high.
>
> Jun
>
> On Fri, May 11, 2012 at 7:13 AM, S Ahmed <[EMAIL PROTECTED]> wrote:
>
> > I was thinking (and after doing some tests on dedicated and ec2), would
> you
> > still say kafka is io intensive?
> >
> > Considering writes are batched every x seconds, and you have a single
> kafka
> > server on a given instance, and consumers are just streaming the data in
> > sequential order (the disk head isn't jumping around), is it safe to say
> > kafka isn't that io intensive to the point that running it on ec2 should
> be
> > just as good as dedicated hardware?
> >
> > I was getting pretty good results on ec2 so this thought came to me...
> >
>
+
Jun Rao 2012-05-11, 16:10
+
S Ahmed 2012-05-11, 16:16
+
Jay Kreps 2012-05-11, 23:10
+
S Ahmed 2012-05-12, 01:21