Kafka, mail # user - Re: only one ProducerSendThread thread when running with multiple brokers (kafka 0.8) - 2014-01-02, 05:41
 Search Hadoop and all its subprojects:

Switch to Threaded View
Copy link to this message
-
Re: only one ProducerSendThread thread when running with multiple brokers (kafka 0.8)
When the producer send thread sends a batch of messages, it first
determines which partition each message should go to. It then groups
messages by broker (based on the leader of the partition of each
message) and sends a produce request per broker (each request may include
multiple partitions). Those produce requests are sent serially. So, if
there is only one partition, only 1 produce request needs to be sent per
batch of messages. If there are 3 partitions, chances are 3 produce
requests are needed. Because those produce requests are sent serially, the
more partitions you have, the more produce requests and the longer the
latency for sending a batch of messages. However, having 12 partitions
shouldn't be significantly worse than 3 partitions since there are only 3
brokers. One way to improve performance is to use a larger batch size. Try
making the batch size 3 times larger with 3 partitions.

Thanks,

Jun
On Wed, Jan 1, 2014 at 12:43 PM, yosi botzer <[EMAIL PROTECTED]> wrote:
 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB