Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Kafka Broker Configuration Tuning and Repartitioning topic


Copy link to this message
-
Re: Kafka Broker Configuration Tuning and Repartitioning topic
Muthu,

a) Not as of now. Please feel free to create the JIRA and specify the
details there

b) I doubt increasing partitions will help. 500 GB/day/topic suggests
the data per partition is only 10 GB/day. Before thinking about
increasing the # of partitions, I would try a few things-

1. Inspect the consumer throughput metrics through the mbeans exposed
on the Kafka consumers.
2. If individual consumer throughput looks reasonable, then deploy
more consumer instances and see if that helps. Since you have 40-50
partitions per topic, you can have at least those many consumer
instances.
3. If not, then check if the consumers post-process the data consumed
from these partitions. If this processing is slow, your consumption
rate will reduce.

Thanks,
Neha

On Tue, Nov 20, 2012 at 3:12 AM, Muthukumar <[EMAIL PROTECTED]> wrote:
> Hi Jun,
>
> Thanks for the response.
>
> a) Is there any plan in the roadmap to address this re-partition or
> partition balance with new partitions? Please let me know to have the
> JIRA for this.
>
> b) Do we need to go for more partitions for the topic6 (46 to ??) to
> reduce the new requests + backlog.
>
> -Muthu
>
> On Tue, Nov 20, 2012 at 11:09 AM, Jun Rao <[EMAIL PROTECTED]> wrote:
>> The configs look reasonable. Currently, we don't repartition existing data.
>> Only new messages will consider the newly added partitions.
>>
>> Thanks
>>
>> Jun
>>