Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> More Kafka Benchmarking Goodness

Copy link to this message
Re: More Kafka Benchmarking Goodness
Can you please share Kafka version you've used for the tests?
On Thu, May 23, 2013 at 8:57 PM, Jason Weiss <[EMAIL PROTECTED]> wrote:

> Folks,
> As I posted to the group here yesterday, my 3 server test in AWS produced
> an average of 273,132 events per second with a fixed-size 2K message
> payload. (Please see that thread for details.) In order to determine the
> horizontal scalability, I added an additional server and MORE producer
> clients into the mix. With 3 servers in the cluster, the average was 91,044
> eps per server (273,132). After adding a 4th server to the mix, and an
> additional set of producers, I achieved an extended run of 361,320 events
> per second with the same fixed-size 2K message payload. The new server
> added an additional 88,188 events per second throughput capacity to the
> cluster – near perfect horizontal scaling.
> While there may be a limit, achieving 500,00 or 750,000 or even 1 million
> events per second with a Kafka cluster isn't beyond reach for the truly
> motivated.
> Nice work committers, very nice work.
> Jason
> This electronic message contains information which may be confidential or
> privileged. The information is intended for the use of the individual or
> entity named above. If you are not the intended recipient, be aware that
> any disclosure, copying, distribution or use of the contents of this
> information is prohibited. If you have received this electronic
> transmission in error, please notify us by e-mail at (
> [EMAIL PROTECTED]) immediately.