These are the performance #'s I got on a dedicated server, everthing
(zookeeper, kafka server) is running on the same server:

bin/kafka-producer-perf-test.sh --brokerinfo broker.list=0:localhost:9092
--messages 1000000 --topic test --threads 10 --message-size=30720
[2012-05-05 14:03:31,344] INFO Creating sync producer for broker id = 0 at
localhost:9092 (kafka.producer.ProducerPool)
[2012-05-05 14:03:31,345] INFO Creating sync producer for broker id = 0 at
localhost:9092 (kafka.producer.ProducerPool)
[2012-05-05 14:03:31,346] INFO Creating sync producer for broker id = 0 at
localhost:9092 (kafka.producer.ProducerPool)
[2012-05-05 14:03:31,346] INFO Creating sync producer for broker id = 0 at
localhost:9092 (kafka.producer.ProducerPool)
[2012-05-05 14:03:31,346] INFO Creating sync producer for broker id = 0 at
localhost:9092 (kafka.producer.ProducerPool)
[2012-05-05 14:03:31,347] INFO Creating sync producer for broker id = 0 at
localhost:9092 (kafka.producer.ProducerPool)
[2012-05-05 14:03:31,347] INFO Creating sync producer for broker id = 0 at
localhost:9092 (kafka.producer.ProducerPool)
[2012-05-05 14:03:31,348] INFO Creating sync producer for broker id = 0 at
localhost:9092 (kafka.producer.ProducerPool)
[2012-05-05 14:03:31,352] INFO Creating sync producer for broker id = 0 at
localhost:9092 (kafka.producer.ProducerPool)
[2012-05-05 14:03:31,353] INFO Messages per thread = 500
(kafka.tools.ProducerPerformance$SyncProducerThread)
[2012-05-05 14:03:31,354] INFO Messages per thread = 500
(kafka.tools.ProducerPerformance$SyncProducerThread)
[2012-05-05 14:03:31,354] INFO Messages per thread = 500
(kafka.tools.ProducerPerformance$SyncProducerThread)
[2012-05-05 14:03:31,354] INFO Messages per thread = 500
(kafka.tools.ProducerPerformance$SyncProducerThread)
[2012-05-05 14:03:31,354] INFO Messages per thread = 500
(kafka.tools.ProducerPerformance$SyncProducerThread)
[2012-05-05 14:03:31,354] INFO Messages per thread = 500
(kafka.tools.ProducerPerformance$SyncProducerThread)
[2012-05-05 14:03:31,354] INFO Messages per thread = 500
(kafka.tools.ProducerPerformance$SyncProducerThread)
[2012-05-05 14:03:31,354] INFO Messages per thread = 500
(kafka.tools.ProducerPerformance$SyncProducerThread)
[2012-05-05 14:03:31,354] INFO Creating sync producer for broker id = 0 at
localhost:9092 (kafka.producer.ProducerPool)
[2012-05-05 14:03:31,357] INFO Messages per thread = 500
(kafka.tools.ProducerPerformance$SyncProducerThread)
[2012-05-05 14:03:31,358] INFO Messages per thread = 500
(kafka.tools.ProducerPerformance$SyncProducerThread)
[2012-05-05 14:03:31,632] INFO Connected to localhost:9092 for producing
(kafka.producer.SyncProducer)
[2012-05-05 14:03:31,632] INFO Connected to localhost:9092 for producing
(kafka.producer.SyncProducer)
[2012-05-05 14:03:31,632] INFO Connected to localhost:9092 for producing
(kafka.producer.SyncProducer)
[2012-05-05 14:03:31,632] INFO Connected to localhost:9092 for producing
(kafka.producer.SyncProducer)
[2012-05-05 14:03:31,632] INFO Connected to localhost:9092 for producing
(kafka.producer.SyncProducer)
[2012-05-05 14:03:31,632] INFO Connected to localhost:9092 for producing
(kafka.producer.SyncProducer)
[2012-05-05 14:03:31,632] INFO Connected to localhost:9092 for producing
(kafka.producer.SyncProducer)
[2012-05-05 14:03:31,632] INFO Connected to localhost:9092 for producing
(kafka.producer.SyncProducer)
[2012-05-05 14:03:31,632] INFO Connected to localhost:9092 for producing
(kafka.producer.SyncProducer)
[2012-05-05 14:03:31,632] INFO Connected to localhost:9092 for producing
(kafka.producer.SyncProducer)
[2012-05-05 14:07:48,431] INFO Closing all sync producers
(kafka.producer.ProducerPool)
[2012-05-05 14:07:48,431] INFO Disconnecting from localhost:9092
(kafka.producer.SyncProducer)
[2012-05-05 14:07:48,885] INFO Closing all sync producers
(kafka.producer.ProducerPool)
[2012-05-05 14:07:48,885] INFO Disconnecting from localhost:9092
(kafka.producer.SyncProducer)
[2012-05-05 14:07:50,600] INFO Closing all sync producers
(kafka.producer.ProducerPool)
[2012-05-05 14:07:50,600] INFO Disconnecting from localhost:9092
(kafka.producer.SyncProducer)
[2012-05-05 14:07:53,427] INFO Closing all sync producers
(kafka.producer.ProducerPool)
[2012-05-05 14:07:53,427] INFO Disconnecting from localhost:9092
(kafka.producer.SyncProducer)
[2012-05-05 14:07:53,429] INFO Closing all sync producers
(kafka.producer.ProducerPool)
[2012-05-05 14:07:53,429] INFO Disconnecting from localhost:9092
(kafka.producer.SyncProducer)
[2012-05-05 14:07:55,643] INFO Closing all sync producers
(kafka.producer.ProducerPool)
[2012-05-05 14:07:55,643] INFO Disconnecting from localhost:9092
(kafka.producer.SyncProducer)
[2012-05-05 14:08:21,173] INFO Closing all sync producers
(kafka.producer.ProducerPool)
[2012-05-05 14:08:21,173] INFO Disconnecting from localhost:9092
(kafka.producer.SyncProducer)
[2012-05-05 14:08:23,758] INFO Closing all sync producers
(kafka.producer.ProducerPool)
[2012-05-05 14:08:23,758] INFO Disconnecting from localhost:9092
(kafka.producer.SyncProducer)
[2012-05-05 14:08:23,922] INFO Closing all sync producers
(kafka.producer.ProducerPool)
[2012-05-05 14:08:23,922] INFO Disconnecting from localhost:9092
(kafka.producer.SyncProducer)
[2012-05-05 14:08:23,952] INFO Closing all sync producers
(kafka.producer.ProducerPool)
[2012-05-05 14:08:23,952] INFO Disconnecting from localhost:9092
(kafka.producer.SyncProducer)
[2012-05-05 14:08:23,953] INFO Total Num Messages: 1000000 bytes:
30720000000 in 292.713 secs (kafka.tools.ProducerPerformance$)
[2012-05-05 14:08:23,981] INFO Messages/sec: 3416.3156
(kafka.tools.ProducerPerformance$)
[2012-05-05 14:08:23,981] INFO MB/sec: 100.0874
(kafka.tools.ProducerPerformance$)

I also noticed when running top, that the java process was pretty much at
100% and my 4GB of ram were used up.
When I tried with a message size of 100KB it crashed with a OOM error.

Any comments on how I could tweak the configuration to get better results?
 My use case

Actually I was thinking, since my payloads sizes are larger, it would make
sense to flush to disk more often right?  Since I only have 4GB ram on the
server.

What effect does the poll time to check if a flush is required have?    Is
it an expensive operation, i.e. does it walk the entire collection and
determine the size?

On Sat, May 5, 2012 at 3:22 PM, S Ahmed <[EMAIL PROTECTED]> wrote:

> These are the performance #'s I got on a dedicated server, everthing
> (zookeeper, kafka server) is running on the same server:
>
> bin/kafka-producer-perf-test.sh --brokerinfo broker.list=0:localhost:9092
> --messages 1000000 --topic test --threads 10 --message-size=30720
> [2012-05-05 14:03:31,344] INFO Creating sync producer for broker id = 0 at
> localhost:9092 (kafka.producer.ProducerPool)
> [2012-05-05 14:03:31,345] INFO Creating sync producer for broker id = 0 at
> localhost:9092 (kafka.producer.ProducerPool)
> [2012-05-05 14:03:31,346] INFO Creating sync producer for broker id = 0 at
> localhost:9092 (kafka.producer.ProducerPool)
> [2012-05-05 14:03:31,346] INFO Creating sync producer for broker id = 0 at
> localhost:9092 (kafka.producer.ProducerPool)
> [2012-05-05 14:03:31,346] INFO Creating sync producer for broker id = 0 at
> localhost:9092 (kafka.producer.ProducerPool)
> [2012-05-05 14:03:31,347] INFO Creating sync producer for broker id = 0 at
> localhost:9092 (kafka.producer.ProducerPool)
> [2012-05-05 14:03:31,347] INFO Creating sync producer for broker id = 0 at
> localhost:9092 (kafka.producer.ProducerPool)
> [2012-05-05 14:03:31,348] INFO Creating sync producer for broker id = 0 at
> localhost:9092 (kafka.producer.ProducerPool)
> [2012-05-05 14:03:31,352] INFO Creating sync producer for broker id = 0 at
> localhost:9092 (kafka.producer.ProducerPool)
> [2012-05-05 14:03:31,353] INFO Messages per thread = 500
> (kafka.tools.ProducerPerformance$SyncProducerThread)
> [2012-05-05 14:03:31,354] INFO Messages per thread = 500
> (kafka.tools.ProducerPerformance$SyncProducerThread)
> [2012-05-05 14:03:31,354] INFO Messages per thread = 500
> (kafka.tools.ProducerPerformance$SyncProducerThread)
> [2012-05-05 14:03:31,354] INFO Messages per thread = 500
> (kafka.tools.ProducerPerformance$SyncProducerThread)
> [2012-05-05 14:03:31,354] INFO Messages per thread = 500
> (kafka.tools.ProducerPerformance$SyncProducerThread)
> [2012-05-05 14:03:31,354] INFO Messages per thread = 500
> (kafka.tools.ProducerPerformance$SyncProducerThread)
> [2012-05-05 14:03:31,354] INFO Messages per thread = 500
> (kafka.tools.ProducerPerformance$SyncProducerThread)
> [2012-05-05 14:03:31,354] INFO Messages per thread = 500
> (kafka.tools.ProducerPerformance$SyncProducerThread)
> [2012-05-05 14:03:31,354] INFO Creating sync producer for broker id = 0 at
> localhost:9092 (kafka.producer.ProducerPool)
> [2012-05-05 14:03:31,357] INFO Messages per thread = 500
> (kafka.tools.ProducerPerformance$SyncProducerThread)
> [2012-05-05 14:03:31,358] INFO Messages per thread = 500
> (kafka.tools.ProducerPerformance$SyncProducerThread)
> [2012-05-05 14:03:31,632] INFO Connected to localhost:9092 for producing
> (kafka.producer.SyncProducer)
> [2012-05-05 14:03:31,632] INFO Connected to localhost:9092 for producing
> (kafka.producer.SyncProducer)
> [2012-05-05 14:03:31,632] INFO Connected to localhost:9092 for producing
> (kafka.producer.SyncProducer)
> [2012-05-05 14:03:31,632] INFO Connected to localhost:9092 for producing
> (kafka.producer.SyncProducer)
> [2012-05-05 14:03:31,632] INFO Connected to localhost:9092 for producing
> (kafka.producer.SyncProducer)
> [2012-05-05 14:03:31,632] INFO Connected to localhost:9092 for producing
> (kafka.producer.SyncProducer)
> [2012-05-05 14:03:31,632] INFO Connected to localhost:9092 for producing
> (kafka.producer.SyncProducer)
> [2012-05-05 14:03:31,632] INFO Connected to localhost:9092 for producing
> (kafka.producer.SyncProducer)
> [2012-05-05 14:03:31,632] INFO Connected to localhost:9092 for producing
> (kafka.producer.SyncProducer)

You can tune log.flush.interval. See
http://incubator.apache.org/kafka/configuration.html for its meaning.

Thanks,

Jun

On Sat, May 5, 2012 at 12:22 PM, S Ahmed <[EMAIL PROTECTED]> wrote:

> These are the performance #'s I got on a dedicated server, everthing
> (zookeeper, kafka server) is running on the same server:
>
> bin/kafka-producer-perf-test.sh --brokerinfo broker.list=0:localhost:9092
> --messages 1000000 --topic test --threads 10 --message-size=30720
> [2012-05-05 14:03:31,344] INFO Creating sync producer for broker id = 0 at
> localhost:9092 (kafka.producer.ProducerPool)
> [2012-05-05 14:03:31,345] INFO Creating sync producer for broker id = 0 at
> localhost:9092 (kafka.producer.ProducerPool)
> [2012-05-05 14:03:31,346] INFO Creating sync producer for broker id = 0 at
> localhost:9092 (kafka.producer.ProducerPool)
> [2012-05-05 14:03:31,346] INFO Creating sync producer for broker id = 0 at
> localhost:9092 (kafka.producer.ProducerPool)
> [2012-05-05 14:03:31,346] INFO Creating sync producer for broker id = 0 at
> localhost:9092 (kafka.producer.ProducerPool)
> [2012-05-05 14:03:31,347] INFO Creating sync producer for broker id = 0 at
> localhost:9092 (kafka.producer.ProducerPool)
> [2012-05-05 14:03:31,347] INFO Creating sync producer for broker id = 0 at
> localhost:9092 (kafka.producer.ProducerPool)
> [2012-05-05 14:03:31,348] INFO Creating sync producer for broker id = 0 at
> localhost:9092 (kafka.producer.ProducerPool)
> [2012-05-05 14:03:31,352] INFO Creating sync producer for broker id = 0 at
> localhost:9092 (kafka.producer.ProducerPool)
> [2012-05-05 14:03:31,353] INFO Messages per thread = 500
> (kafka.tools.ProducerPerformance$SyncProducerThread)
> [2012-05-05 14:03:31,354] INFO Messages per thread = 500
> (kafka.tools.ProducerPerformance$SyncProducerThread)
> [2012-05-05 14:03:31,354] INFO Messages per thread = 500
> (kafka.tools.ProducerPerformance$SyncProducerThread)
> [2012-05-05 14:03:31,354] INFO Messages per thread = 500
> (kafka.tools.ProducerPerformance$SyncProducerThread)
> [2012-05-05 14:03:31,354] INFO Messages per thread = 500
> (kafka.tools.ProducerPerformance$SyncProducerThread)
> [2012-05-05 14:03:31,354] INFO Messages per thread = 500
> (kafka.tools.ProducerPerformance$SyncProducerThread)
> [2012-05-05 14:03:31,354] INFO Messages per thread = 500
> (kafka.tools.ProducerPerformance$SyncProducerThread)
> [2012-05-05 14:03:31,354] INFO Messages per thread = 500
> (kafka.tools.ProducerPerformance$SyncProducerThread)
> [2012-05-05 14:03:31,354] INFO Creating sync producer for broker id = 0 at
> localhost:9092 (kafka.producer.ProducerPool)
> [2012-05-05 14:03:31,357] INFO Messages per thread = 500
> (kafka.tools.ProducerPerformance$SyncProducerThread)
> [2012-05-05 14:03:31,358] INFO Messages per thread = 500
> (kafka.tools.ProducerPerformance$SyncProducerThread)
> [2012-05-05 14:03:31,632] INFO Connected to localhost:9092 for producing
> (kafka.producer.SyncProducer)
> [2012-05-05 14:03:31,632] INFO Connected to localhost:9092 for producing
> (kafka.producer.SyncProducer)
> [2012-05-05 14:03:31,632] INFO Connected to localhost:9092 for producing
> (kafka.producer.SyncProducer)
> [2012-05-05 14:03:31,632] INFO Connected to localhost:9092 for producing
> (kafka.producer.SyncProducer)
> [2012-05-05 14:03:31,632] INFO Connected to localhost:9092 for producing
> (kafka.producer.SyncProducer)
> [2012-05-05 14:03:31,632] INFO Connected to localhost:9092 for producing
> (kafka.producer.SyncProducer)
> [2012-05-05 14:03:31,632] INFO Connected to localhost:9092 for producing
> (kafka.producer.SyncProducer)
> [2012-05-05 14:03:31,632] INFO Connected to localhost:9092 for producing
> (kafka.producer.SyncProducer)
> [2012-05-05 14:03:31,632] INFO Connected to localhost:9092 for producing
> (kafka.producer.SyncProducer)
> [2012-05-05 14:03:31,632] INFO Connected to localhost:9092 for producing
> (kafka.producer.SyncProducer)
> [2012-05-05 14:07:48,431] INFO Closing all sync producers
> (kafka.producer.ProducerPool)