Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume, mail # dev - spooldir source reading Flume itself and thinking the file has changed (1.3.1)


+
Edward Sargisson 2013-05-08, 16:04
Copy link to this message
-
Re: spooldir source reading Flume itself and thinking the file has changed (1.3.1)
Mike Percy 2013-05-08, 17:23
Hi Edward,
Spooldir assumes that the file is atomically moved into the directory in
question, maybe we should make it wait around until he last mod time is >
10 seconds or something to avoid situations where a readdir   + stat
catches it during a file copy operation and thinks you're changing the
files. If you're sure that the file name is not being reused then maybe
it's a bug.

We require file name to be unique because we don't have a great way to read
the inode with Java.

Mike
On Wed, May 8, 2013 at 9:04 AM, Edward Sargisson <[EMAIL PROTECTED]> wrote:

> Hi all,
> I've been experimenting with using Flume to transport Flume's own logs.
>
> My setup is that I have the apache-log4j-extras RollingFileAppender rolling
> the file every minute into a spool directory. Then I have Flume with the
> spooldir source reading that directory.
>
> Very often I see:
> java.lang.IllegalStateException: File name has been re-used with different
> files. Spooling assumpti
> on violated for
> /opt/castellan/log/spool/castellan-reader.20130430T2055.log.COMPLETED
>         at
>
> org.apache.flume.client.avro.SpoolingFileLineReader.retireCurrentFile(SpoolingFileLineRe
> ader.java:272)
>
> Then it never recovers, repeatedly saying:
> 2013-05-08 15:57:19,233 ERROR [pool-10-thread-1]
> org.apache.flume.source.SpoolDirectorySource Uncau
> ght exception in Runnable
> java.io.IOException: Stream closed
>         at java.io.BufferedReader.ensureOpen(BufferedReader.java:115)
>         at java.io.BufferedReader.readLine(BufferedReader.java:310)
>         at java.io.BufferedReader.readLine(BufferedReader.java:382)
>         at
>
> org.apache.flume.client.avro.SpoolingFileLineReader.readLines(SpoolingFileLineReader.java:180)
>
> I note that 1.4 has extensive changes to the spooldir source but nothing
> that really seemed to speak to this.
>
> Does anybody else see this? Anything I should look for? Or is this some
> kind of race condition between log4j and spooldir?
>
> Cheers,
> Edward
>
+
Edward Sargisson 2013-05-11, 00:02
+
Mike Percy 2013-05-11, 01:08
+
Edward Sargisson 2013-05-13, 15:45
+
Edward Sargisson 2013-05-14, 23:50
+
Mike Percy 2013-05-22, 07:18
+
Edward Sargisson 2013-05-22, 16:22
+
Phil Scala 2013-05-23, 00:24
+
Mike Percy 2013-05-23, 01:13