Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka, mail # user - Kafka Broker Configuration Tuning and Repartitioning topic


Copy link to this message
-
Re: Kafka Broker Configuration Tuning and Repartitioning topic
Jun Rao 2012-11-20, 05:39
The configs look reasonable. Currently, we don't repartition existing data.
Only new messages will consider the newly added partitions.

Thanks

Jun

On Mon, Nov 19, 2012 at 9:34 AM, Muthukumar <[EMAIL PROTECTED]> wrote:

> Hi,
>
> We're currently having 1 broker to 10 consumers setup pointing to 5 ZK
> node. Load would be in 500GB data / day, and following is the generic
> configuration we're using as,
>
> brokerid=0
> port=9092
> num.threads=8
> socket.send.buffer=1048576
> socket.receive.buffer=1048576
> max.socket.request.bytes=104857600
> log.dir=/data/kafka/server/logs
> num.partitions=16
>
> topic.partition.count.map=topic1:16,topic2:16,topic3:16,topic4:16,topic5:16,topic6:16,topic7:46,topic8:16
> log.flush.interval=10000
> log.default.flush.interval.ms=1000
> log.default.flush.scheduler.interval.ms=1000
> log.retention.hours=48
> log.file.size=536870912
> log.cleanup.interval.mins=1
> enable.zookeeper=true
> zk.connectiontimeout.ms=1000000
>
> 1) Please let us know if this configuration is holds good or needs a
> change?
> 2) We're seeing a huge backlog of 320GB for topic6 partitioned for 46.
> Is it possible to re-partition since 21 partition were newly added and
> data lag is almost 0 there.
>
> Let me know if more details are required. And thanks for your great help.
>
> -Muthu
>