Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Kafka broker not respecting log.roll.hours?


Copy link to this message
-
Re: Kafka broker not respecting log.roll.hours?
Someone pointed out a particularly easy fix: don't reuse files after a
restart. Done. I really like that. Simple. Any chance of this happening any
time soon?
On Sun, Apr 28, 2013 at 2:04 AM, Swapnil Ghike <[EMAIL PROTECTED]> wrote:

> @Dan: Upon restart of the broker, if a segment already has data, the
> broker resets the firstAppendTime of the segment to the time when that
> segment's file handles are being loaded into memory. Thus as you correctly
> explained, every time you shut down a broker, the broker essentially
> forgets the firstAppendTime. This behavior is present in both 0.7.2 and
> 0.8.
>
> As Jun said, ideally we should set firstAppendTime to the file creation
> time. Unfortunately Java nio can provide you provide you that information
> only if the underlying filesystem implementation supports the notion of
> file creation time.
>
> Thanks for filing the JIRA, these are good suggestions.
>
> @Jason: Thanks for pointing out that log.roll.hours is not documented on
> the website. This config was added late in 0.7 and we probably forgot to
> update the website. We have filed KAFKA-834/KAFKA-835 to update the
> configs and other documentation on the website in general. Please let us
> know if you see any other missing piece.
>
> Thanks,
> Swapnil
>
> On 4/27/13 2:36 PM, "Dan Frankowski" <[EMAIL PROTECTED]> wrote:
>
> >I believe there is a separate watcher thread. The only issue is upon
> >restart the broker forgets when the file was created. The behavior I
> >described (files can be appended to infinitely) is awkward for us. We have
> >tried to work around it.
> >
> >
> >On Fri, Apr 26, 2013 at 10:32 AM, Adam Talaat <[EMAIL PROTECTED]> wrote:
> >
> >> I don't know how Kafka's rollover algorithm is implemented, but this is
> >> common behavior for other logging frameworks. You would need a separate
> >> watcher/scheduled thread to rollover a log file, even if no events were
> >> coming in. Logback (and probably log4j, by the same author) dispenses
> >>with
> >> the watcher thread. Instead, it checks each message as it comes in and
> >> decides whether the message should belong in a new file. If it should, a
> >> rollover of the old file is triggered and the message is deposited in
> >>the
> >> new file. But no rollover will occur until a message that belongs in a
> >>new
> >> file arrives.
> >>
> >> Cheers,
> >> Adam
> >>
> >>
> >>
> >> On Fri, Apr 26, 2013 at 9:52 AM, Jason Rosenberg <[EMAIL PROTECTED]>
> >>wrote:
> >>
> >> > By the way, is there a reason why 'log.roll.hours' is not documented
> >>on
> >> the
> >> > apache configuration page:
> >>http://kafka.apache.org/configuration.html ?
> >> >
> >> > It's possible to find this setting (and several other undocumented
> >> > settings) by looking at the source code.  I'm just not sure why the
> >> > complete set of options is not documented on the site (is it meant to
> >>be
> >> > experimental?).
> >> >
> >> > Jason
> >> >
> >> >
> >> > On Fri, Apr 26, 2013 at 8:19 AM, Dan Frankowski <[EMAIL PROTECTED]>
> >> > wrote:
> >> >
> >> > > https://issues.apache.org/jira/browse/KAFKA-881
> >> > >
> >> > > Thanks.
> >> > >
> >> > >
> >> > > On Fri, Apr 26, 2013 at 7:40 AM, Jun Rao <[EMAIL PROTECTED]> wrote:
> >> > >
> >> > > > Yes, for low volume topic, the time-based rolling can be
> >>imprecise.
> >> > Could
> >> > > > you file a jira and describe your suggestions there? Ideally, we
> >> should
> >> > > set
> >> > > > firstAppendTime to the file creation time. However, it doesn't
> >>seem
> >> you
> >> > > can
> >> > > > get the creation time in java.
> >> > > >
> >> > > > Thanks,
> >> > > >
> >> > > > Jun
> >> > > >
> >> > > >
> >> > > > On Thu, Apr 25, 2013 at 11:12 PM, Dan Frankowski
> >><[EMAIL PROTECTED]
> >> >
> >> > > > wrote:
> >> > > >
> >> > > > > We have high-volume topics and low-volume topics. The problem
> >> occurs
> >> > > more
> >> > > > > often for low-volume topics to be sure.
> >> > > > >
> >> > > > > But if my hypothesis is correct about why it is happening, here

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB