Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> my performance numbers


Copy link to this message
-
my performance numbers
These are the performance #'s I got on a dedicated server, everthing
(zookeeper, kafka server) is running on the same server:

bin/kafka-producer-perf-test.sh --brokerinfo broker.list=0:localhost:9092
--messages 1000000 --topic test --threads 10 --message-size=30720
[2012-05-05 14:03:31,344] INFO Creating sync producer for broker id = 0 at
localhost:9092 (kafka.producer.ProducerPool)
[2012-05-05 14:03:31,345] INFO Creating sync producer for broker id = 0 at
localhost:9092 (kafka.producer.ProducerPool)
[2012-05-05 14:03:31,346] INFO Creating sync producer for broker id = 0 at
localhost:9092 (kafka.producer.ProducerPool)
[2012-05-05 14:03:31,346] INFO Creating sync producer for broker id = 0 at
localhost:9092 (kafka.producer.ProducerPool)
[2012-05-05 14:03:31,346] INFO Creating sync producer for broker id = 0 at
localhost:9092 (kafka.producer.ProducerPool)
[2012-05-05 14:03:31,347] INFO Creating sync producer for broker id = 0 at
localhost:9092 (kafka.producer.ProducerPool)
[2012-05-05 14:03:31,347] INFO Creating sync producer for broker id = 0 at
localhost:9092 (kafka.producer.ProducerPool)
[2012-05-05 14:03:31,348] INFO Creating sync producer for broker id = 0 at
localhost:9092 (kafka.producer.ProducerPool)
[2012-05-05 14:03:31,352] INFO Creating sync producer for broker id = 0 at
localhost:9092 (kafka.producer.ProducerPool)
[2012-05-05 14:03:31,353] INFO Messages per thread = 500
(kafka.tools.ProducerPerformance$SyncProducerThread)
[2012-05-05 14:03:31,354] INFO Messages per thread = 500
(kafka.tools.ProducerPerformance$SyncProducerThread)
[2012-05-05 14:03:31,354] INFO Messages per thread = 500
(kafka.tools.ProducerPerformance$SyncProducerThread)
[2012-05-05 14:03:31,354] INFO Messages per thread = 500
(kafka.tools.ProducerPerformance$SyncProducerThread)
[2012-05-05 14:03:31,354] INFO Messages per thread = 500
(kafka.tools.ProducerPerformance$SyncProducerThread)
[2012-05-05 14:03:31,354] INFO Messages per thread = 500
(kafka.tools.ProducerPerformance$SyncProducerThread)
[2012-05-05 14:03:31,354] INFO Messages per thread = 500
(kafka.tools.ProducerPerformance$SyncProducerThread)
[2012-05-05 14:03:31,354] INFO Messages per thread = 500
(kafka.tools.ProducerPerformance$SyncProducerThread)
[2012-05-05 14:03:31,354] INFO Creating sync producer for broker id = 0 at
localhost:9092 (kafka.producer.ProducerPool)
[2012-05-05 14:03:31,357] INFO Messages per thread = 500
(kafka.tools.ProducerPerformance$SyncProducerThread)
[2012-05-05 14:03:31,358] INFO Messages per thread = 500
(kafka.tools.ProducerPerformance$SyncProducerThread)
[2012-05-05 14:03:31,632] INFO Connected to localhost:9092 for producing
(kafka.producer.SyncProducer)
[2012-05-05 14:03:31,632] INFO Connected to localhost:9092 for producing
(kafka.producer.SyncProducer)
[2012-05-05 14:03:31,632] INFO Connected to localhost:9092 for producing
(kafka.producer.SyncProducer)
[2012-05-05 14:03:31,632] INFO Connected to localhost:9092 for producing
(kafka.producer.SyncProducer)
[2012-05-05 14:03:31,632] INFO Connected to localhost:9092 for producing
(kafka.producer.SyncProducer)
[2012-05-05 14:03:31,632] INFO Connected to localhost:9092 for producing
(kafka.producer.SyncProducer)
[2012-05-05 14:03:31,632] INFO Connected to localhost:9092 for producing
(kafka.producer.SyncProducer)
[2012-05-05 14:03:31,632] INFO Connected to localhost:9092 for producing
(kafka.producer.SyncProducer)
[2012-05-05 14:03:31,632] INFO Connected to localhost:9092 for producing
(kafka.producer.SyncProducer)
[2012-05-05 14:03:31,632] INFO Connected to localhost:9092 for producing
(kafka.producer.SyncProducer)
[2012-05-05 14:07:48,431] INFO Closing all sync producers
(kafka.producer.ProducerPool)
[2012-05-05 14:07:48,431] INFO Disconnecting from localhost:9092
(kafka.producer.SyncProducer)
[2012-05-05 14:07:48,885] INFO Closing all sync producers
(kafka.producer.ProducerPool)
[2012-05-05 14:07:48,885] INFO Disconnecting from localhost:9092
(kafka.producer.SyncProducer)
[2012-05-05 14:07:50,600] INFO Closing all sync producers
(kafka.producer.ProducerPool)
[2012-05-05 14:07:50,600] INFO Disconnecting from localhost:9092
(kafka.producer.SyncProducer)
[2012-05-05 14:07:53,427] INFO Closing all sync producers
(kafka.producer.ProducerPool)
[2012-05-05 14:07:53,427] INFO Disconnecting from localhost:9092
(kafka.producer.SyncProducer)
[2012-05-05 14:07:53,429] INFO Closing all sync producers
(kafka.producer.ProducerPool)
[2012-05-05 14:07:53,429] INFO Disconnecting from localhost:9092
(kafka.producer.SyncProducer)
[2012-05-05 14:07:55,643] INFO Closing all sync producers
(kafka.producer.ProducerPool)
[2012-05-05 14:07:55,643] INFO Disconnecting from localhost:9092
(kafka.producer.SyncProducer)
[2012-05-05 14:08:21,173] INFO Closing all sync producers
(kafka.producer.ProducerPool)
[2012-05-05 14:08:21,173] INFO Disconnecting from localhost:9092
(kafka.producer.SyncProducer)
[2012-05-05 14:08:23,758] INFO Closing all sync producers
(kafka.producer.ProducerPool)
[2012-05-05 14:08:23,758] INFO Disconnecting from localhost:9092
(kafka.producer.SyncProducer)
[2012-05-05 14:08:23,922] INFO Closing all sync producers
(kafka.producer.ProducerPool)
[2012-05-05 14:08:23,922] INFO Disconnecting from localhost:9092
(kafka.producer.SyncProducer)
[2012-05-05 14:08:23,952] INFO Closing all sync producers
(kafka.producer.ProducerPool)
[2012-05-05 14:08:23,952] INFO Disconnecting from localhost:9092
(kafka.producer.SyncProducer)
[2012-05-05 14:08:23,953] INFO Total Num Messages: 1000000 bytes:
30720000000 in 292.713 secs (kafka.tools.ProducerPerformance$)
[2012-05-05 14:08:23,981] INFO Messages/sec: 3416.3156
(kafka.tools.ProducerPerformance$)
[2012-05-05 14:08:23,981] INFO MB/sec: 100.0874
(kafka.tools.ProducerPerformance$)

I also noticed when running top, that the java process was pretty much at
100% and my 4GB of ram were used up.
When I tried with a message size of 100KB it crashed with a OOM error.

Any comments on how I could tweak the configuration to get better results?
 My use case
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB