Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> questtion about log.retention.bytes


Copy link to this message
-
Re: questtion about log.retention.bytes
For the first question, yes, topic C will get the value of 30MB.

For the second question, log.retention.bytes only controls the segment log
file size, not the index. Typically, index file size is much smaller than
the log file. The index file of the last (active) segment is presized to
the max index size (defaults to 10MB). However, the size is trimmed as soon
as the segment rolls.

Thanks,

Jun
On Mon, Aug 19, 2013 at 9:22 AM, Paul Christian
<[EMAIL PROTECTED]>wrote:

> Hi Jun,
>
> Thank you for your reply. I'm still a little fuzzy on the concept.
>
> Are you saying I can have topic A, B and C and with
>
> log.retention.bytes.per.topic.A = 15MB
> log.retention.bytes.per.topic.B = 20MB
> log.retention.bytes = 30MB
>
> And thus topic C will get the value 30MB? Since it's not defined like the
> others' 'per topic'?
>
> log.retention.bytes is for all topics that are not included in
> log.retention.bytes.per.topic
> (which defines a map of topic -> size).
>
> Otherwise, log.retention.bytes.per.topic and log.retention.bytes seem very
> similar to me.
>
> Additionally, we've experimented with this value on our test cluster where
> we set the log.retention.bytes to 11MB as a test. Below is a snippet from
> our server.properties.
>
> # A size-based retention policy for logs. Segments are pruned from the log
> as long as the remaining
> # segments don't drop below log.retention.bytes.
> log.retention.bytes=11534336
>
> Here is a ls -lh from one of the topics
>
> -rw-r--r-- 1 kafka service  10M Aug 19 15:45 00000000000007021913.index
> -rw-r--r-- 1 kafka service 114M Aug 19 15:45 00000000000007021913.log
>
> The index file appears to be reflected in the property
> log.index.size.max.bytes, but the log just keeps going.
>

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB