Kafka, mail # user - Re: Kafka producer behavior - 2013-12-18, 10:45
 Search Hadoop and all its subprojects:

Switch to Threaded View
Copy link to this message
Re: Kafka producer behavior

this is a gotcha about kafka producer partitioning, you much send the
messages with a non null key.
If the key is null kafka will not call the partitioner.

Because with this partitioner the key does not matter you can pass in a
constant string like "1" etc.

Oh one more thing, on performance:

The produce's send method has a synchronized block on the producer
instance, which means performance goes down the drain.
I could only get (on a 12 core, 72 gig ram) machine 13K tps out of the
producer. A way to solve this is to instantiate an array/list of N
producers and then in your send code round robin over the producers.
I got to 80K tps (for my use case) using 6 producer instances from a single
box sending to 3 kafka servers.

send ( msg ) {
  producers[ producer-index.getAndIncrement() % producer_count ].send(msg)

On Wed, Dec 18, 2013 at 11:24 AM, Hanish Bansal <
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB