Edward Sargisson 2013-05-08, 16:04
Mike Percy 2013-05-08, 17:23
Edward Sargisson 2013-05-11, 00:02
-Re: spooldir source reading Flume itself and thinking the file has changed (1.3.1)
Mike Percy 2013-05-11, 01:08
Thanks for investigating. I'm definitely open to suggestions for
improvement with this. Maybe dying is a bit extreme… the goal was to ensure
that people could not possibly try to use it to tail a file, which will
definitely not work correctly! :)
On Fri, May 10, 2013 at 5:02 PM, Edward Sargisson <[EMAIL PROTECTED]> wrote:
> Hi Mike,
> I was curious so I went on a bit of a hunt through logger source code.
> The result is that loggers can't be relied on to atomically roll the
> file so a feature to allow a delay before checking the file would be
> of great utility.
> For that matter, having Flume not die completely in this scenario
> would also be good.
> apache-log4j-extras does this :
> return source.renameTo(destination);
> logback does this :
> boolean result = srcFile.renameTo(targetFile);
> log4j2 does this :
> srcStream = new FileInputStream(source);
> destStream = new FileOutputStream(destination);
> srcChannel = srcStream.getChannel();
> destChannel = destStream.getChannel();
> destChannel.transferFrom(srcChannel, 0, srcChannel.size());
> The JavaDoc for File.renameTo says:
> Many aspects of the behavior of this method are inherently
> platform-dependent: The rename operation might not be able to move a
> file from one filesystem to another, it might not be atomic, and it
> might not succeed if a file with the destination abstract pathname
> already exists. The return value should always be checked to make sure
> that the rename operation was successful.
> My conclusion is that the loggers (except possibly log4j2) can't be
> relied on to atomically roll the file.
> >Hi Edward,
> >Spooldir assumes that the file is atomically moved into the directory in
> >question, maybe we should make it wait around until he last mod time is >
> >10 seconds or something to avoid situations where a readdir + stat
> >catches it during a file copy operation and thinks you're changing the
> >files. If you're sure that the file name is not being reused then maybe
> >it's a bug.
> >We require file name to be unique because we don't have a great way to
> >the inode with Java.
> >On Wed, May 8, 2013 at 9:04 AM, Edward Sargisson <[EMAIL PROTECTED]>
> > Hi all,
> > I've been experimenting with using Flume to transport Flume's own logs.
> > My setup is that I have the apache-log4j-extras RollingFileAppender
> > the file every minute into a spool directory. Then I have Flume with the
> > spooldir source reading that directory.
> > Very often I see:
> > java.lang.IllegalStateException: File name has been re-used with
> > files. Spooling assumpti
> > on violated for
> > /opt/castellan/log/spool/castellan-reader.20130430T2055.log.COMPLETED
> > at
> > ader.java:272)
> > Then it never recovers, repeatedly saying:
> > 2013-05-08 15:57:19,233 ERROR [pool-10-thread-1]
> > org.apache.flume.source.SpoolDirectorySource Uncau
> > ght exception in Runnable
> > java.io.IOException: Stream closed
> > at java.io.BufferedReader.ensureOpen(BufferedReader.java:115)
> > at java.io.BufferedReader.readLine(BufferedReader.java:310)
> > at java.io.BufferedReader.readLine(BufferedReader.java:382)
> > at
Edward Sargisson 2013-05-13, 15:45
Edward Sargisson 2013-05-14, 23:50
Mike Percy 2013-05-22, 07:18
Edward Sargisson 2013-05-22, 16:22
Phil Scala 2013-05-23, 00:24
Mike Percy 2013-05-23, 01:13