According to the Kafka 8 documentation under broker configuration. There are these parameters and their definitions.
log.retention.bytes -1 The maximum size of the log before deleting it log.retention.bytes.per.topic "" The maximum size of the log for some specific topic before deleting it
I'm curious what the first value 'log.retention.bytes' is for if the second one is for per topic logs, because aren't all logs generated per topic? Is this an aggregate value across topics?
Related question, is there a parameter for kafka where you can say only hold this much TOTAL data across all topic ( logs/index together )? I.e. our hosts have this much available space and so value log.retention.whatever.aggregate == 75% total disk space.
For the first question, yes, topic C will get the value of 30MB.
For the second question, log.retention.bytes only controls the segment log file size, not the index. Typically, index file size is much smaller than the log file. The index file of the last (active) segment is presized to the max index size (defaults to 10MB). However, the size is trimmed as soon as the segment rolls.
Jun On Mon, Aug 19, 2013 at 9:22 AM, Paul Christian <[EMAIL PROTECTED]>wrote:
I'm trying to understand the 2nd problem you reported. Are you saying that you set the log.retention.bytes=11534336 (11MB) but nevertheless your log grew to 114MB. Which means the config option didn't really work as expected?
Thanks, Neha On Mon, Aug 19, 2013 at 8:46 PM, Jun Rao <[EMAIL PROTECTED]> wrote:
I think the problem is that there is no way to understand the meaning of that config from the docs, so people keep asking over and over again. The docs make iit sound like it is per topic and just says it is the "maximum size before it is deleted" which makes no sense...
Jay made a good point. The docs says it is for the topic but actually it is for one partition. Thanks, Jun.
NEW: Monitor These Apps!
Apache Lucene, Apache Solr and all other Apache Software Foundation project and their respective logos are trademarks of the Apache Software Foundation.
Elasticsearch, Kibana, Logstash, and Beats are trademarks of Elasticsearch BV, registered in the U.S. and in other countries. This site and Sematext Group is in no way affiliated with Elasticsearch BV.
Service operated by Sematext