Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka, mail # user - Kafka broker not respecting log.roll.hours?


+
Dan Frankowski 2013-04-25, 19:45
+
Jun Rao 2013-04-26, 04:49
+
Dan Frankowski 2013-04-26, 06:13
+
Jun Rao 2013-04-26, 14:40
+
Dan Frankowski 2013-04-26, 15:20
+
Jason Rosenberg 2013-04-26, 16:52
+
Adam Talaat 2013-04-26, 17:33
+
Dan Frankowski 2013-04-27, 21:37
Copy link to this message
-
Re: Kafka broker not respecting log.roll.hours?
Dan Frankowski 2013-05-02, 21:23
Someone pointed out a particularly easy fix: don't reuse files after a
restart. Done. I really like that. Simple. Any chance of this happening any
time soon?
On Sun, Apr 28, 2013 at 2:04 AM, Swapnil Ghike <[EMAIL PROTECTED]> wrote:

> @Dan: Upon restart of the broker, if a segment already has data, the
> broker resets the firstAppendTime of the segment to the time when that
> segment's file handles are being loaded into memory. Thus as you correctly
> explained, every time you shut down a broker, the broker essentially
> forgets the firstAppendTime. This behavior is present in both 0.7.2 and
> 0.8.
>
> As Jun said, ideally we should set firstAppendTime to the file creation
> time. Unfortunately Java nio can provide you provide you that information
> only if the underlying filesystem implementation supports the notion of
> file creation time.
>
> Thanks for filing the JIRA, these are good suggestions.
>
> @Jason: Thanks for pointing out that log.roll.hours is not documented on
> the website. This config was added late in 0.7 and we probably forgot to
> update the website. We have filed KAFKA-834/KAFKA-835 to update the
> configs and other documentation on the website in general. Please let us
> know if you see any other missing piece.
>
> Thanks,
> Swapnil
>
> On 4/27/13 2:36 PM, "Dan Frankowski" <[EMAIL PROTECTED]> wrote:
>
> >I believe there is a separate watcher thread. The only issue is upon
> >restart the broker forgets when the file was created. The behavior I
> >described (files can be appended to infinitely) is awkward for us. We have
> >tried to work around it.
> >
> >
> >On Fri, Apr 26, 2013 at 10:32 AM, Adam Talaat <[EMAIL PROTECTED]> wrote:
> >
> >> I don't know how Kafka's rollover algorithm is implemented, but this is
> >> common behavior for other logging frameworks. You would need a separate
> >> watcher/scheduled thread to rollover a log file, even if no events were
> >> coming in. Logback (and probably log4j, by the same author) dispenses
> >>with
> >> the watcher thread. Instead, it checks each message as it comes in and
> >> decides whether the message should belong in a new file. If it should, a
> >> rollover of the old file is triggered and the message is deposited in
> >>the
> >> new file. But no rollover will occur until a message that belongs in a
> >>new
> >> file arrives.
> >>
> >> Cheers,
> >> Adam
> >>
> >>
> >>
> >> On Fri, Apr 26, 2013 at 9:52 AM, Jason Rosenberg <[EMAIL PROTECTED]>
> >>wrote:
> >>
> >> > By the way, is there a reason why 'log.roll.hours' is not documented
> >>on
> >> the
> >> > apache configuration page:
> >>http://kafka.apache.org/configuration.html ?
> >> >
> >> > It's possible to find this setting (and several other undocumented
> >> > settings) by looking at the source code.  I'm just not sure why the
> >> > complete set of options is not documented on the site (is it meant to
> >>be
> >> > experimental?).
> >> >
> >> > Jason
> >> >
> >> >
> >> > On Fri, Apr 26, 2013 at 8:19 AM, Dan Frankowski <[EMAIL PROTECTED]>
> >> > wrote:
> >> >
> >> > > https://issues.apache.org/jira/browse/KAFKA-881
> >> > >
> >> > > Thanks.
> >> > >
> >> > >
> >> > > On Fri, Apr 26, 2013 at 7:40 AM, Jun Rao <[EMAIL PROTECTED]> wrote:
> >> > >
> >> > > > Yes, for low volume topic, the time-based rolling can be
> >>imprecise.
> >> > Could
> >> > > > you file a jira and describe your suggestions there? Ideally, we
> >> should
> >> > > set
> >> > > > firstAppendTime to the file creation time. However, it doesn't
> >>seem
> >> you
> >> > > can
> >> > > > get the creation time in java.
> >> > > >
> >> > > > Thanks,
> >> > > >
> >> > > > Jun
> >> > > >
> >> > > >
> >> > > > On Thu, Apr 25, 2013 at 11:12 PM, Dan Frankowski
> >><[EMAIL PROTECTED]
> >> >
> >> > > > wrote:
> >> > > >
> >> > > > > We have high-volume topics and low-volume topics. The problem
> >> occurs
> >> > > more
> >> > > > > often for low-volume topics to be sure.
> >> > > > >
> >> > > > > But if my hypothesis is correct about why it is happening, here

 
+
Jun Rao 2013-05-03, 15:58