Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka, mail # user - questtion about log.retention.bytes


Copy link to this message
-
Re: questtion about log.retention.bytes
Jun Rao 2013-08-28, 15:02
Changed the wording in the doc.

Thanks,

Jun
On Wed, Aug 28, 2013 at 7:20 AM, Jay Kreps <[EMAIL PROTECTED]> wrote:

> I think the problem is that there is no way to understand the meaning of
> that config from the docs, so people keep asking over and over again. The
> docs make iit sound like it is per topic and just says it is the "maximum
> size before it is deleted" which makes no sense...
>
> -jay
>
> On Tuesday, August 27, 2013, Jun Rao wrote:
>
> > For the first question, yes.
> >
> > For the second one, this is documented in
> > http://kafka.apache.org/documentation.html#brokerconfigs
> >
> > "Note that all per topic configuration properties below have the format
> of
> > csv (e.g., "topic1:value1,topic2:value2")."
> > Thanks,
> > Jun
> >
> >
> >
> > On Tue, Aug 27, 2013 at 11:52 AM, Yu, Libo <[EMAIL PROTECTED]
> <javascript:;>>
> > wrote:
> >
> > > Hi Jun,
> > >
> > > In a previous email thread
> > >
> >
> http://markmail.org/search/?q=kafka+log.retention.bytes#query:kafka%20log.retention.bytes+page:1+mid:qnt4pbq47goii2ui+state:results
> > > ,
> > > you said log.retention.bytes is for each partition. Could you clarify
> on
> > > that?
> > >
> > > Say if I have a topic with three partitions. I want to limit the disk
> > > space to 1Gb for each partition.
> > > Then log.retention.bytes should be set to 1Gb (not 3Gb). Is that right?
> > >
> > > If I want to use log.retention.bytes.per.topic to set the same limit,
> > > should it be set to 1G or 3G?
> > >
> > > Libo
> > >
> > >
> > > -----Original Message-----
> > > From: Jun Rao [mailto:[EMAIL PROTECTED] <javascript:;>]
> > > Sent: Saturday, August 17, 2013 12:40 AM
> > > To: [EMAIL PROTECTED] <javascript:;>
> > > Subject: Re: questtion about log.retention.bytes
> > >
> > > log.retention.bytes is for all topics that are not included in
> > > log.retention.bytes.per.topic (which defines a map of topic -> size).
> > >
> > > Currently, we don't have a total size limit across all topics.
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > >
> > > On Fri, Aug 16, 2013 at 2:00 PM, Paul Christian
> > > <[EMAIL PROTECTED] <javascript:;>>wrote:
> > >
> > > > According to the Kafka 8 documentation under broker configuration.
> > > > There are these parameters and their definitions.
> > > >
> > > > log.retention.bytes -1 The maximum size of the log before deleting it
> > > > log.retention.bytes.per.topic "" The maximum size of the log for some
> > > > specific topic before deleting it
> > > >
> > > > I'm curious what the first value 'log.retention.bytes' is for if the
> > > > second one is for per topic logs, because aren't all logs generated
> > > > per topic? Is this an aggregate value across topics?
> > > >
> > > > Related question, is there a parameter for kafka where you can say
> > > > only hold this much TOTAL data across all topic ( logs/index together
> > )?
> > > I.e.
> > > > our  hosts have this much available space and so value
> > > > log.retention.whatever.aggregate == 75% total  disk space.
> > > >
> > >
> >
>