Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> since flushes are batched, is it still io intensive?

Copy link to this message
Re: since flushes are batched, is it still io intensive?
What about memory?  I know you guys have 24GB of ram per server?

Basically I'm juggling between going with a dedicated box (which has faster
IO), or ec2 which has slower IO but cheaper on the ram side (way cheaper!).

On Fri, May 11, 2012 at 10:34 AM, Jun Rao <[EMAIL PROTECTED]> wrote:

> It all depends on the volume of the data. At LinkedIn, we observed that the
> io load on a typical Kafka broker is not high.
> Jun
> On Fri, May 11, 2012 at 7:13 AM, S Ahmed <[EMAIL PROTECTED]> wrote:
> > I was thinking (and after doing some tests on dedicated and ec2), would
> you
> > still say kafka is io intensive?
> >
> > Considering writes are batched every x seconds, and you have a single
> kafka
> > server on a given instance, and consumers are just streaming the data in
> > sequential order (the disk head isn't jumping around), is it safe to say
> > kafka isn't that io intensive to the point that running it on ec2 should
> be
> > just as good as dedicated hardware?
> >
> > I was getting pretty good results on ec2 so this thought came to me...
> >