Home | About | Sematext search-lucene.com search-hadoop.com
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Analysis of producer performance


Copy link to this message
-
Re: Analysis of producer performance -- and Producer-Kafka reliability
Philip,

We would not use spooling to local disk on the producer to deal with
problems with the connection to the brokers, but rather to absorb temporary
spikes in traffic that would overwhelm the brokers. This is assuming that
1) those spikes are relatively short, but when they come they require much
higher throughput than normal (otherwise we'd just have a capacity problem
and would need more brokers), and 2) the spikes are long enough for just a
RAM buffer to be dangerous. If the brokers did go down, spooling to disk
would give us more time to react, but that's not the primary reason for
wanting the feature.

-Piotr

On Fri, Apr 12, 2013 at 8:21 AM, Philip O'Toole <[EMAIL PROTECTED]> wrote:
 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB