Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Kafka Broker Configuration Tuning and Repartitioning topic


Copy link to this message
-
Re: Kafka Broker Configuration Tuning and Repartitioning topic
The configs look reasonable. Currently, we don't repartition existing data.
Only new messages will consider the newly added partitions.

Thanks

Jun

On Mon, Nov 19, 2012 at 9:34 AM, Muthukumar <[EMAIL PROTECTED]> wrote:

> Hi,
>
> We're currently having 1 broker to 10 consumers setup pointing to 5 ZK
> node. Load would be in 500GB data / day, and following is the generic
> configuration we're using as,
>
> brokerid=0
> port=9092
> num.threads=8
> socket.send.buffer=1048576
> socket.receive.buffer=1048576
> max.socket.request.bytes=104857600
> log.dir=/data/kafka/server/logs
> num.partitions=16
>
> topic.partition.count.map=topic1:16,topic2:16,topic3:16,topic4:16,topic5:16,topic6:16,topic7:46,topic8:16
> log.flush.interval=10000
> log.default.flush.interval.ms=1000
> log.default.flush.scheduler.interval.ms=1000
> log.retention.hours=48
> log.file.size=536870912
> log.cleanup.interval.mins=1
> enable.zookeeper=true
> zk.connectiontimeout.ms=1000000
>
> 1) Please let us know if this configuration is holds good or needs a
> change?
> 2) We're seeing a huge backlog of 320GB for topic6 partitioned for 46.
> Is it possible to re-partition since 21 partition were newly added and
> data lag is almost 0 there.
>
> Let me know if more details are required. And thanks for your great help.
>
> -Muthu
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB