Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # dev >> spooldir source reading Flume itself and thinking the file has changed (1.3.1)

Copy link to this message
Re: spooldir source reading Flume itself and thinking the file has changed (1.3.1)
Hi Phil,
Since this is more of a dev discussion I'll just continue the conversation
here on this list.

FYI the latest Spool Directory Source has support for resuming reading
files. Trunk / Flume 1.4 have some new code around this aspect.

Regarding Linux, doing lsof is a pretty cool idea but not portable to all
systems. Also in Linux, two processes are allowed to have the same file
open for write (it's still a bad idea though). I don't know of a portable
way to check whether some other process have a given file open. We could,
however, check to see if the file changed, and if so just stop processing
that file for a while and try again later. I just don't want people to
think spooldir is good for "tailing" a file, because it's not… :)


On Wed, May 22, 2013 at 5:24 PM, Phil Scala <[EMAIL PROTECTED]>wrote:

> Hey Ed / Mike
> Besides a comment in the users mailing list that I commented on the file
> spool starting from the beginning of the file if there was a failure.  The
> code does have that well commented (in the retireCurrentFile) [if you don't
> retire the file then you run the risk of duplicates...fine with my use :)]
> As Ed mentioned we have been chatting about ensuring there are no
> invariants muddled up during file spool processing.  I see this as 2 or 3
> pieces...I think the code is pretty solid, with one area I want to look
> into.
> I would like to give this more thought...
> The file the spool source has decided is the "next file"... is it in
> use/has the "upload" to the spool directory completed.
>         Discussions mentioned some "time" delay -> that could be
> artificial and still never solve the problem.   I need to do some learning
> here, coming from windows the file locking was pretty exclusive.  I want to
> see about FileChannel locks in nio and Linux file management.    This could
> maybe be an area to look at.  Right now there are no locks obtained for the
> file being processed.
> I will come back with something a little better formulated soon...
> Thanks
> Phil Scala
> Software Developer / Architect
> Global Relay
> 866.484.6630  |  [EMAIL PROTECTED]  |  globalrelay.com
> -----Original Message-----
> From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]] On Behalf Of Edward
> Sargisson
> Sent: Wednesday, May 22, 2013 12:22 PM
> To: Mike Percy; [EMAIL PROTECTED]
> Subject: Re: spooldir source reading Flume itself and thinking the file
> has changed (1.3.1)
> Hi Mike,
> I haven't tried log4j2 in my environments but my review of the log4j2
> change is that it should work.
> What would I change?
> Phil Scala may have some thoughts.
> It would be nice if we thought through the file locking. I want to be able
> to put a file in the spooldir and know that Flume isn't going to get
> started until I'm ready. This certainly involves thinking about what the
> file-putting process is doing but it's not clear to me how to ensure this
> whole part is safe.
> The thing that is currently annoying is handling stack traces. All logging
> systems I've seen (except recent log4j2) output the stack trace with each
> frame on a new line. This means that each frame gets its own log event and
> the timestamp has to be added by Flume (instead of taken from the original
> event). That Flume timestamp might be delayed by up to 1 minute (because of
> log rolling so its pretty crap). Logstash has a multiline filter that
> somewhat solves this.
> My current approach is to try and get the Log4j2 FlumeAppender and Flume
> 1.3.1 reliable and trustworthy.
> Cheers,
> Edward
> "Hi Edward,
> Did the fixes in LOG4J2-254 fix your file rolling issue?
> What are your thoughts on how to improve spooling directory source's error
> handling when it detects a change in the file? Just bail and retry later? I
> suppose that's a pretty reasonable approach.
> Regards,
> Mike
> On Tue, May 14, 2013 at 4:50 PM, Edward Sargisson <[EMAIL PROTECTED]