Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka, mail # user - questtion about log.retention.bytes


Copy link to this message
-
Re: questtion about log.retention.bytes
Jun Rao 2013-08-28, 03:54
For the first question, yes.

For the second one, this is documented in
http://kafka.apache.org/documentation.html#brokerconfigs

"Note that all per topic configuration properties below have the format of
csv (e.g., "topic1:value1,topic2:value2")."
Thanks,
Jun

On Tue, Aug 27, 2013 at 11:52 AM, Yu, Libo <[EMAIL PROTECTED]> wrote:

> Hi Jun,
>
> In a previous email thread
> http://markmail.org/search/?q=kafka+log.retention.bytes#query:kafka%20log.retention.bytes+page:1+mid:qnt4pbq47goii2ui+state:results
> ,
> you said log.retention.bytes is for each partition. Could you clarify on
> that?
>
> Say if I have a topic with three partitions. I want to limit the disk
> space to 1Gb for each partition.
> Then log.retention.bytes should be set to 1Gb (not 3Gb). Is that right?
>
> If I want to use log.retention.bytes.per.topic to set the same limit,
> should it be set to 1G or 3G?
>
> Libo
>
>
> -----Original Message-----
> From: Jun Rao [mailto:[EMAIL PROTECTED]]
> Sent: Saturday, August 17, 2013 12:40 AM
> To: [EMAIL PROTECTED]
> Subject: Re: questtion about log.retention.bytes
>
> log.retention.bytes is for all topics that are not included in
> log.retention.bytes.per.topic (which defines a map of topic -> size).
>
> Currently, we don't have a total size limit across all topics.
>
> Thanks,
>
> Jun
>
>
> On Fri, Aug 16, 2013 at 2:00 PM, Paul Christian
> <[EMAIL PROTECTED]>wrote:
>
> > According to the Kafka 8 documentation under broker configuration.
> > There are these parameters and their definitions.
> >
> > log.retention.bytes -1 The maximum size of the log before deleting it
> > log.retention.bytes.per.topic "" The maximum size of the log for some
> > specific topic before deleting it
> >
> > I'm curious what the first value 'log.retention.bytes' is for if the
> > second one is for per topic logs, because aren't all logs generated
> > per topic? Is this an aggregate value across topics?
> >
> > Related question, is there a parameter for kafka where you can say
> > only hold this much TOTAL data across all topic ( logs/index together )?
> I.e.
> > our  hosts have this much available space and so value
> > log.retention.whatever.aggregate == 75% total  disk space.
> >
>