As I posted to the group here yesterday, my 3 server test in AWS produced an average of 273,132 events per second with a fixed-size 2K message payload. (Please see that thread for details.) In order to determine the horizontal scalability, I added an additional server and MORE producer clients into the mix. With 3 servers in the cluster, the average was 91,044 eps per server (273,132). After adding a 4th server to the mix, and an additional set of producers, I achieved an extended run of 361,320 events per second with the same fixed-size 2K message payload. The new server added an additional 88,188 events per second throughput capacity to the cluster – near perfect horizontal scaling.
While there may be a limit, achieving 500,00 or 750,000 or even 1 million events per second with a Kafka cluster isn't beyond reach for the truly motivated.
Nice work committers, very nice work.
This electronic message contains information which may be confidential or privileged. The information is intended for the use of the individual or entity named above. If you are not the intended recipient, be aware that any disclosure, copying, distribution or use of the contents of this information is prohibited. If you have received this electronic transmission in error, please notify us by e-mail at ([EMAIL PROTECTED]) immediately.
I'm not sure how those values will hold up for 0.8, at least in the initial release. Would love to see your performance analysis in a short blog post :)
Thanks, Neha On Thu, May 23, 2013 at 1:24 PM, Oleg Ruchovets <[EMAIL PROTECTED]>wrote:
NEW: Monitor These Apps!
Apache Lucene, Apache Solr and all other Apache Software Foundation project and their respective logos are trademarks of the Apache Software Foundation.
Elasticsearch, Kibana, Logstash, and Beats are trademarks of Elasticsearch BV, registered in the U.S. and in other countries. This site and Sematext Group is in no way affiliated with Elasticsearch BV.
Service operated by Sematext