Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> questtion about log.retention.bytes


Copy link to this message
-
Re: questtion about log.retention.bytes
For the first question, yes.

For the second one, this is documented in
http://kafka.apache.org/documentation.html#brokerconfigs

"Note that all per topic configuration properties below have the format of
csv (e.g., "topic1:value1,topic2:value2")."
Thanks,
Jun

On Tue, Aug 27, 2013 at 11:52 AM, Yu, Libo <[EMAIL PROTECTED]> wrote:

> Hi Jun,
>
> In a previous email thread
> http://markmail.org/search/?q=kafka+log.retention.bytes#query:kafka%20log.retention.bytes+page:1+mid:qnt4pbq47goii2ui+state:results
> ,
> you said log.retention.bytes is for each partition. Could you clarify on
> that?
>
> Say if I have a topic with three partitions. I want to limit the disk
> space to 1Gb for each partition.
> Then log.retention.bytes should be set to 1Gb (not 3Gb). Is that right?
>
> If I want to use log.retention.bytes.per.topic to set the same limit,
> should it be set to 1G or 3G?
>
> Libo
>
>
> -----Original Message-----
> From: Jun Rao [mailto:[EMAIL PROTECTED]]
> Sent: Saturday, August 17, 2013 12:40 AM
> To: [EMAIL PROTECTED]
> Subject: Re: questtion about log.retention.bytes
>
> log.retention.bytes is for all topics that are not included in
> log.retention.bytes.per.topic (which defines a map of topic -> size).
>
> Currently, we don't have a total size limit across all topics.
>
> Thanks,
>
> Jun
>
>
> On Fri, Aug 16, 2013 at 2:00 PM, Paul Christian
> <[EMAIL PROTECTED]>wrote:
>
> > According to the Kafka 8 documentation under broker configuration.
> > There are these parameters and their definitions.
> >
> > log.retention.bytes -1 The maximum size of the log before deleting it
> > log.retention.bytes.per.topic "" The maximum size of the log for some
> > specific topic before deleting it
> >
> > I'm curious what the first value 'log.retention.bytes' is for if the
> > second one is for per topic logs, because aren't all logs generated
> > per topic? Is this an aggregate value across topics?
> >
> > Related question, is there a parameter for kafka where you can say
> > only hold this much TOTAL data across all topic ( logs/index together )?
> I.e.
> > our  hosts have this much available space and so value
> > log.retention.whatever.aggregate == 75% total  disk space.
> >
>

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB