Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> questtion about log.retention.bytes


Copy link to this message
-
Re: questtion about log.retention.bytes
Changed the wording in the doc.

Thanks,

Jun
On Wed, Aug 28, 2013 at 7:20 AM, Jay Kreps <[EMAIL PROTECTED]> wrote:

> I think the problem is that there is no way to understand the meaning of
> that config from the docs, so people keep asking over and over again. The
> docs make iit sound like it is per topic and just says it is the "maximum
> size before it is deleted" which makes no sense...
>
> -jay
>
> On Tuesday, August 27, 2013, Jun Rao wrote:
>
> > For the first question, yes.
> >
> > For the second one, this is documented in
> > http://kafka.apache.org/documentation.html#brokerconfigs
> >
> > "Note that all per topic configuration properties below have the format
> of
> > csv (e.g., "topic1:value1,topic2:value2")."
> > Thanks,
> > Jun
> >
> >
> >
> > On Tue, Aug 27, 2013 at 11:52 AM, Yu, Libo <[EMAIL PROTECTED]
> <javascript:;>>
> > wrote:
> >
> > > Hi Jun,
> > >
> > > In a previous email thread
> > >
> >
> http://markmail.org/search/?q=kafka+log.retention.bytes#query:kafka%20log.retention.bytes+page:1+mid:qnt4pbq47goii2ui+state:results
> > > ,
> > > you said log.retention.bytes is for each partition. Could you clarify
> on
> > > that?
> > >
> > > Say if I have a topic with three partitions. I want to limit the disk
> > > space to 1Gb for each partition.
> > > Then log.retention.bytes should be set to 1Gb (not 3Gb). Is that right?
> > >
> > > If I want to use log.retention.bytes.per.topic to set the same limit,
> > > should it be set to 1G or 3G?
> > >
> > > Libo
> > >
> > >
> > > -----Original Message-----
> > > From: Jun Rao [mailto:[EMAIL PROTECTED] <javascript:;>]
> > > Sent: Saturday, August 17, 2013 12:40 AM
> > > To: [EMAIL PROTECTED] <javascript:;>
> > > Subject: Re: questtion about log.retention.bytes
> > >
> > > log.retention.bytes is for all topics that are not included in
> > > log.retention.bytes.per.topic (which defines a map of topic -> size).
> > >
> > > Currently, we don't have a total size limit across all topics.
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > >
> > > On Fri, Aug 16, 2013 at 2:00 PM, Paul Christian
> > > <[EMAIL PROTECTED] <javascript:;>>wrote:
> > >
> > > > According to the Kafka 8 documentation under broker configuration.
> > > > There are these parameters and their definitions.
> > > >
> > > > log.retention.bytes -1 The maximum size of the log before deleting it
> > > > log.retention.bytes.per.topic "" The maximum size of the log for some
> > > > specific topic before deleting it
> > > >
> > > > I'm curious what the first value 'log.retention.bytes' is for if the
> > > > second one is for per topic logs, because aren't all logs generated
> > > > per topic? Is this an aggregate value across topics?
> > > >
> > > > Related question, is there a parameter for kafka where you can say
> > > > only hold this much TOTAL data across all topic ( logs/index together
> > )?
> > > I.e.
> > > > our  hosts have this much available space and so value
> > > > log.retention.whatever.aggregate == 75% total  disk space.
> > > >
> > >
> >
>

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB