What trouble shooting steps can we do to identify if bottleneck is with
consuming or producing..? Does changing anything in log4j configuration or
a jmx mbeans provide insight into it..? Does Metadata refresh interval
affect picking up new partitions for only existing topic or it affect
picking up any new topics..?
From: Jun Rao <[EMAIL PROTECTED]>
Date: Fri, Aug 23, 2013 at 12:08 AM
Subject: Re: Tuning mirror maker performance
To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>
You have to determine whether the bottleneck is in the consumer or the
To improve the performance of the latter, you can increase the # of total
consumer streams. # streams is capped by total # partitions. So, you may
need to increase the # of partitions.
To improve the performance of the latter, you can (a) increase the batch
size in async mode and/or (b) run more instances of producers.
Metadata refresh interval is configurable. It's mainly for the producer to
pick up newly available partitions.
On Thu, Aug 22, 2013 at 1:44 PM, Rajasekar Elango <[EMAIL PROTECTED]https://cwiki.apache.org/confluence/display/KAFKA/Kafka+mirroring+(MirrorMaker)#Kafkamirroring%28MirrorMaker%29-Consumerandsourceclustersocketbuffersizes