Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume >> mail # dev >> spooldir source reading Flume itself and thinking the file has changed (1.3.1)


Copy link to this message
-
spooldir source reading Flume itself and thinking the file has changed (1.3.1)
Hi all,
I've been experimenting with using Flume to transport Flume's own logs.

My setup is that I have the apache-log4j-extras RollingFileAppender rolling
the file every minute into a spool directory. Then I have Flume with the
spooldir source reading that directory.

Very often I see:
java.lang.IllegalStateException: File name has been re-used with different
files. Spooling assumpti
on violated for
/opt/castellan/log/spool/castellan-reader.20130430T2055.log.COMPLETED
        at
org.apache.flume.client.avro.SpoolingFileLineReader.retireCurrentFile(SpoolingFileLineRe
ader.java:272)

Then it never recovers, repeatedly saying:
2013-05-08 15:57:19,233 ERROR [pool-10-thread-1]
org.apache.flume.source.SpoolDirectorySource Uncau
ght exception in Runnable
java.io.IOException: Stream closed
        at java.io.BufferedReader.ensureOpen(BufferedReader.java:115)
        at java.io.BufferedReader.readLine(BufferedReader.java:310)
        at java.io.BufferedReader.readLine(BufferedReader.java:382)
        at
org.apache.flume.client.avro.SpoolingFileLineReader.readLines(SpoolingFileLineReader.java:180)

I note that 1.4 has extensive changes to the spooldir source but nothing
that really seemed to speak to this.

Does anybody else see this? Anything I should look for? Or is this some
kind of race condition between log4j and spooldir?

Cheers,
Edward
+
Mike Percy 2013-05-08, 17:23
+
Edward Sargisson 2013-05-11, 00:02
+
Mike Percy 2013-05-11, 01:08
+
Edward Sargisson 2013-05-13, 15:45
+
Edward Sargisson 2013-05-14, 23:50
+
Mike Percy 2013-05-22, 07:18
+
Edward Sargisson 2013-05-22, 16:22
+
Phil Scala 2013-05-23, 00:24
+
Mike Percy 2013-05-23, 01:13