Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka >> mail # user >> since flushes are batched, is it still io intensive?


+
S Ahmed 2012-05-11, 14:13
+
Jun Rao 2012-05-11, 14:34
Copy link to this message
-
Re: since flushes are batched, is it still io intensive?
What about memory?  I know you guys have 24GB of ram per server?

Basically I'm juggling between going with a dedicated box (which has faster
IO), or ec2 which has slower IO but cheaper on the ram side (way cheaper!).

On Fri, May 11, 2012 at 10:34 AM, Jun Rao <[EMAIL PROTECTED]> wrote:

> It all depends on the volume of the data. At LinkedIn, we observed that the
> io load on a typical Kafka broker is not high.
>
> Jun
>
> On Fri, May 11, 2012 at 7:13 AM, S Ahmed <[EMAIL PROTECTED]> wrote:
>
> > I was thinking (and after doing some tests on dedicated and ec2), would
> you
> > still say kafka is io intensive?
> >
> > Considering writes are batched every x seconds, and you have a single
> kafka
> > server on a given instance, and consumers are just streaming the data in
> > sequential order (the disk head isn't jumping around), is it safe to say
> > kafka isn't that io intensive to the point that running it on ec2 should
> be
> > just as good as dedicated hardware?
> >
> > I was getting pretty good results on ec2 so this thought came to me...
> >
>
+
Jun Rao 2012-05-11, 16:10
+
S Ahmed 2012-05-11, 16:16
+
Jay Kreps 2012-05-11, 23:10
+
S Ahmed 2012-05-12, 01:21
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB