Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka, mail # user - Kafka broker not respecting log.roll.hours?


Copy link to this message
-
Re: Kafka broker not respecting log.roll.hours?
Jun Rao 2013-05-03, 15:58
Commented on the jira.

Thanks,

Jun
On Thu, May 2, 2013 at 2:22 PM, Dan Frankowski <[EMAIL PROTECTED]> wrote:

> Someone pointed out a particularly easy fix: don't reuse files after a
> restart. Done. I really like that. Simple. Any chance of this happening any
> time soon?
>
>
> On Sun, Apr 28, 2013 at 2:04 AM, Swapnil Ghike <[EMAIL PROTECTED]>
> wrote:
>
> > @Dan: Upon restart of the broker, if a segment already has data, the
> > broker resets the firstAppendTime of the segment to the time when that
> > segment's file handles are being loaded into memory. Thus as you
> correctly
> > explained, every time you shut down a broker, the broker essentially
> > forgets the firstAppendTime. This behavior is present in both 0.7.2 and
> > 0.8.
> >
> > As Jun said, ideally we should set firstAppendTime to the file creation
> > time. Unfortunately Java nio can provide you provide you that information
> > only if the underlying filesystem implementation supports the notion of
> > file creation time.
> >
> > Thanks for filing the JIRA, these are good suggestions.
> >
> > @Jason: Thanks for pointing out that log.roll.hours is not documented on
> > the website. This config was added late in 0.7 and we probably forgot to
> > update the website. We have filed KAFKA-834/KAFKA-835 to update the
> > configs and other documentation on the website in general. Please let us
> > know if you see any other missing piece.
> >
> > Thanks,
> > Swapnil
> >
> > On 4/27/13 2:36 PM, "Dan Frankowski" <[EMAIL PROTECTED]> wrote:
> >
> > >I believe there is a separate watcher thread. The only issue is upon
> > >restart the broker forgets when the file was created. The behavior I
> > >described (files can be appended to infinitely) is awkward for us. We
> have
> > >tried to work around it.
> > >
> > >
> > >On Fri, Apr 26, 2013 at 10:32 AM, Adam Talaat <[EMAIL PROTECTED]>
> wrote:
> > >
> > >> I don't know how Kafka's rollover algorithm is implemented, but this
> is
> > >> common behavior for other logging frameworks. You would need a
> separate
> > >> watcher/scheduled thread to rollover a log file, even if no events
> were
> > >> coming in. Logback (and probably log4j, by the same author) dispenses
> > >>with
> > >> the watcher thread. Instead, it checks each message as it comes in and
> > >> decides whether the message should belong in a new file. If it
> should, a
> > >> rollover of the old file is triggered and the message is deposited in
> > >>the
> > >> new file. But no rollover will occur until a message that belongs in a
> > >>new
> > >> file arrives.
> > >>
> > >> Cheers,
> > >> Adam
> > >>
> > >>
> > >>
> > >> On Fri, Apr 26, 2013 at 9:52 AM, Jason Rosenberg <[EMAIL PROTECTED]>
> > >>wrote:
> > >>
> > >> > By the way, is there a reason why 'log.roll.hours' is not documented
> > >>on
> > >> the
> > >> > apache configuration page:
> > >>http://kafka.apache.org/configuration.html ?
> > >> >
> > >> > It's possible to find this setting (and several other undocumented
> > >> > settings) by looking at the source code.  I'm just not sure why the
> > >> > complete set of options is not documented on the site (is it meant
> to
> > >>be
> > >> > experimental?).
> > >> >
> > >> > Jason
> > >> >
> > >> >
> > >> > On Fri, Apr 26, 2013 at 8:19 AM, Dan Frankowski <[EMAIL PROTECTED]
> >
> > >> > wrote:
> > >> >
> > >> > > https://issues.apache.org/jira/browse/KAFKA-881
> > >> > >
> > >> > > Thanks.
> > >> > >
> > >> > >
> > >> > > On Fri, Apr 26, 2013 at 7:40 AM, Jun Rao <[EMAIL PROTECTED]>
> wrote:
> > >> > >
> > >> > > > Yes, for low volume topic, the time-based rolling can be
> > >>imprecise.
> > >> > Could
> > >> > > > you file a jira and describe your suggestions there? Ideally, we
> > >> should
> > >> > > set
> > >> > > > firstAppendTime to the file creation time. However, it doesn't
> > >>seem
> > >> you
> > >> > > can
> > >> > > > get the creation time in java.
> > >> > > >
> > >> > > > Thanks,
> > >> > > >
> > >> > > > Jun
> > >> > > >
> > >> > > >
> > >