Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka, mail # user - my performance numbers


Copy link to this message
-
my performance numbers
S Ahmed 2012-05-05, 19:22
These are the performance #'s I got on a dedicated server, everthing
(zookeeper, kafka server) is running on the same server:

bin/kafka-producer-perf-test.sh --brokerinfo broker.list=0:localhost:9092
--messages 1000000 --topic test --threads 10 --message-size=30720
[2012-05-05 14:03:31,344] INFO Creating sync producer for broker id = 0 at
localhost:9092 (kafka.producer.ProducerPool)
[2012-05-05 14:03:31,345] INFO Creating sync producer for broker id = 0 at
localhost:9092 (kafka.producer.ProducerPool)
[2012-05-05 14:03:31,346] INFO Creating sync producer for broker id = 0 at
localhost:9092 (kafka.producer.ProducerPool)
[2012-05-05 14:03:31,346] INFO Creating sync producer for broker id = 0 at
localhost:9092 (kafka.producer.ProducerPool)
[2012-05-05 14:03:31,346] INFO Creating sync producer for broker id = 0 at
localhost:9092 (kafka.producer.ProducerPool)
[2012-05-05 14:03:31,347] INFO Creating sync producer for broker id = 0 at
localhost:9092 (kafka.producer.ProducerPool)
[2012-05-05 14:03:31,347] INFO Creating sync producer for broker id = 0 at
localhost:9092 (kafka.producer.ProducerPool)
[2012-05-05 14:03:31,348] INFO Creating sync producer for broker id = 0 at
localhost:9092 (kafka.producer.ProducerPool)
[2012-05-05 14:03:31,352] INFO Creating sync producer for broker id = 0 at
localhost:9092 (kafka.producer.ProducerPool)
[2012-05-05 14:03:31,353] INFO Messages per thread = 500
(kafka.tools.ProducerPerformance$SyncProducerThread)
[2012-05-05 14:03:31,354] INFO Messages per thread = 500
(kafka.tools.ProducerPerformance$SyncProducerThread)
[2012-05-05 14:03:31,354] INFO Messages per thread = 500
(kafka.tools.ProducerPerformance$SyncProducerThread)
[2012-05-05 14:03:31,354] INFO Messages per thread = 500
(kafka.tools.ProducerPerformance$SyncProducerThread)
[2012-05-05 14:03:31,354] INFO Messages per thread = 500
(kafka.tools.ProducerPerformance$SyncProducerThread)
[2012-05-05 14:03:31,354] INFO Messages per thread = 500
(kafka.tools.ProducerPerformance$SyncProducerThread)
[2012-05-05 14:03:31,354] INFO Messages per thread = 500
(kafka.tools.ProducerPerformance$SyncProducerThread)
[2012-05-05 14:03:31,354] INFO Messages per thread = 500
(kafka.tools.ProducerPerformance$SyncProducerThread)
[2012-05-05 14:03:31,354] INFO Creating sync producer for broker id = 0 at
localhost:9092 (kafka.producer.ProducerPool)
[2012-05-05 14:03:31,357] INFO Messages per thread = 500
(kafka.tools.ProducerPerformance$SyncProducerThread)
[2012-05-05 14:03:31,358] INFO Messages per thread = 500
(kafka.tools.ProducerPerformance$SyncProducerThread)
[2012-05-05 14:03:31,632] INFO Connected to localhost:9092 for producing
(kafka.producer.SyncProducer)
[2012-05-05 14:03:31,632] INFO Connected to localhost:9092 for producing
(kafka.producer.SyncProducer)
[2012-05-05 14:03:31,632] INFO Connected to localhost:9092 for producing
(kafka.producer.SyncProducer)
[2012-05-05 14:03:31,632] INFO Connected to localhost:9092 for producing
(kafka.producer.SyncProducer)
[2012-05-05 14:03:31,632] INFO Connected to localhost:9092 for producing
(kafka.producer.SyncProducer)
[2012-05-05 14:03:31,632] INFO Connected to localhost:9092 for producing
(kafka.producer.SyncProducer)
[2012-05-05 14:03:31,632] INFO Connected to localhost:9092 for producing
(kafka.producer.SyncProducer)
[2012-05-05 14:03:31,632] INFO Connected to localhost:9092 for producing
(kafka.producer.SyncProducer)
[2012-05-05 14:03:31,632] INFO Connected to localhost:9092 for producing
(kafka.producer.SyncProducer)
[2012-05-05 14:03:31,632] INFO Connected to localhost:9092 for producing
(kafka.producer.SyncProducer)
[2012-05-05 14:07:48,431] INFO Closing all sync producers
(kafka.producer.ProducerPool)
[2012-05-05 14:07:48,431] INFO Disconnecting from localhost:9092
(kafka.producer.SyncProducer)
[2012-05-05 14:07:48,885] INFO Closing all sync producers
(kafka.producer.ProducerPool)
[2012-05-05 14:07:48,885] INFO Disconnecting from localhost:9092
(kafka.producer.SyncProducer)
[2012-05-05 14:07:50,600] INFO Closing all sync producers
(kafka.producer.ProducerPool)
[2012-05-05 14:07:50,600] INFO Disconnecting from localhost:9092
(kafka.producer.SyncProducer)
[2012-05-05 14:07:53,427] INFO Closing all sync producers
(kafka.producer.ProducerPool)
[2012-05-05 14:07:53,427] INFO Disconnecting from localhost:9092
(kafka.producer.SyncProducer)
[2012-05-05 14:07:53,429] INFO Closing all sync producers
(kafka.producer.ProducerPool)
[2012-05-05 14:07:53,429] INFO Disconnecting from localhost:9092
(kafka.producer.SyncProducer)
[2012-05-05 14:07:55,643] INFO Closing all sync producers
(kafka.producer.ProducerPool)
[2012-05-05 14:07:55,643] INFO Disconnecting from localhost:9092
(kafka.producer.SyncProducer)
[2012-05-05 14:08:21,173] INFO Closing all sync producers
(kafka.producer.ProducerPool)
[2012-05-05 14:08:21,173] INFO Disconnecting from localhost:9092
(kafka.producer.SyncProducer)
[2012-05-05 14:08:23,758] INFO Closing all sync producers
(kafka.producer.ProducerPool)
[2012-05-05 14:08:23,758] INFO Disconnecting from localhost:9092
(kafka.producer.SyncProducer)
[2012-05-05 14:08:23,922] INFO Closing all sync producers
(kafka.producer.ProducerPool)
[2012-05-05 14:08:23,922] INFO Disconnecting from localhost:9092
(kafka.producer.SyncProducer)
[2012-05-05 14:08:23,952] INFO Closing all sync producers
(kafka.producer.ProducerPool)
[2012-05-05 14:08:23,952] INFO Disconnecting from localhost:9092
(kafka.producer.SyncProducer)
[2012-05-05 14:08:23,953] INFO Total Num Messages: 1000000 bytes:
30720000000 in 292.713 secs (kafka.tools.ProducerPerformance$)
[2012-05-05 14:08:23,981] INFO Messages/sec: 3416.3156
(kafka.tools.ProducerPerformance$)
[2012-05-05 14:08:23,981] INFO MB/sec: 100.0874
(kafka.tools.ProducerPerformance$)

I also noticed when running top, that the java process was pretty much at
100% and my 4GB of ram were used up.
When I tried with a message size of 100KB it crashed with a OOM error.

Any comments on how I could tweak the configuration to get better results?
 My use case