Home | About | Sematext search-lucene.com search-hadoop.com
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # dev >> Abou Kafka 0.8 producer throughput test


Copy link to this message
-
Re: Abou Kafka 0.8 producer throughput test
This is a good question.

As mentioned we have some experience running this with no ack and there are
a lot of downsides. We considered making the ack optional, but this would
complicate the producer api since we could give back the offset only in the
case where there is an ack.

Thinking about it more we realized there is no real performance hit, just
latency, and you only pay for the latency if you want to wait for the
response. This resulted in the current tentative plan which is to make all
requests async, always return "future response" so you only block if you
want to get the result. This is the best possible end-state since we can
give the rich general api with the same performance as without the ack.

However this requires a fairly large change in the client which we haven't
done yet.

So in 0.8 synchronous producer performance will decrease. Asynchronous
production should probably not be too much worse because the async
production and batching masks and amortizes the latency. It is too early to
say how much worse it will be as there are still a few perf issues to
resolve.

It would be reasonable if people were a little annoyed by this since we are
effectively making the software worse on some dimensions before we make it
better. Our reasoning was that batching up even more changes into a single
release was just too dangerous. People who care about replication will
(hopefully) care enough about this that taking a hit on sync producer
performance will be okay, and people who don't care about replication can
just skip a version since that is the major feature in 0.8.

-Jay
On Tue, Jan 22, 2013 at 7:10 PM, S Ahmed <[EMAIL PROTECTED]> wrote:
 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB