Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Anyone running kafka with a single broker in production? what about only 8GB ram?


Copy link to this message
-
Re: Anyone running kafka with a single broker in production? what about only 8GB ram?
Hello,

In most cases of Kafka, network bottleneck will be hit before the disk
bottleneck. So maybe you want to check your network capacity to see if it
has been saturated.

Guozhang
On Thu, Oct 10, 2013 at 3:57 PM, Bruno D. Rodrigues <
[EMAIL PROTECTED]> wrote:

> A 10/10/2013, às 23:14, S Ahmed <[EMAIL PROTECTED]> escreveu:
>
> > Is anyone out there running a single broker kafka setup?
> >
> > How about with only 8 GB RAM?
> >
> > I'm looking at one of the better dedicated server prodivers, and a 8GB
> > server is pretty much what I want to spend at the moment, would it make
> > sense going this route?
> > This same server would also potentially be running zookeeper also.
> >
> > In terms of messages per second, at most I would be seeing about 2000
> > messages per second, of 20KB to 200KB in size.
> >
> > I know the people at linkedin are running with I believe 24GB of ram.
>
> My personal newbie experience, which is surely completely wrong and
> miss-configured, got me up to 70MB/sec, either with controlled 1K messages
> (hence 70Kmsg/sec) as well as with more random data (test data from 100
> bytes to a couple MB). First I thought the 70MB were the hard disk limit,
> but when I got the same result both with a proper linux server with a 10K
> disk, as well as with a Mac mini with a 5400rpm disk, I got confused.
>
> The mini has 2G, the linux server has 8 or 16, can'r recall at the moment.
>
> The test was performed both with single and multi producers and consumers.
> One producer = 70MB, two producers = 35MB each and so forth. Running
> standalone instances on each server, same value. Running both together in 2
> partition 2 replica crossed mode, same result.
>
> As far as I understood, more memory just means more kernel buffer space to
> speed up the lack of disk speed, as kafka seems to not really depend on
> memory for the queueing.
>
>
>
--
-- Guozhang

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB