Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # dev >> spooldir source reading Flume itself and thinking the file has changed (1.3.1)


Copy link to this message
-
Re: spooldir source reading Flume itself and thinking the file has changed (1.3.1)
Hi Phil,
Since this is more of a dev discussion I'll just continue the conversation
here on this list.

FYI the latest Spool Directory Source has support for resuming reading
files. Trunk / Flume 1.4 have some new code around this aspect.

Regarding Linux, doing lsof is a pretty cool idea but not portable to all
systems. Also in Linux, two processes are allowed to have the same file
open for write (it's still a bad idea though). I don't know of a portable
way to check whether some other process have a given file open. We could,
however, check to see if the file changed, and if so just stop processing
that file for a while and try again later. I just don't want people to
think spooldir is good for "tailing" a file, because it's not… :)

Regards,
Mike

On Wed, May 22, 2013 at 5:24 PM, Phil Scala <[EMAIL PROTECTED]>wrote:

> Hey Ed / Mike
>
> Besides a comment in the users mailing list that I commented on the file
> spool starting from the beginning of the file if there was a failure.  The
> code does have that well commented (in the retireCurrentFile) [if you don't
> retire the file then you run the risk of duplicates...fine with my use :)]
>
>
> As Ed mentioned we have been chatting about ensuring there are no
> invariants muddled up during file spool processing.  I see this as 2 or 3
> pieces...I think the code is pretty solid, with one area I want to look
> into.
>
> I would like to give this more thought...
>
> The file the spool source has decided is the "next file"... is it in
> use/has the "upload" to the spool directory completed.
>
>         Discussions mentioned some "time" delay -> that could be
> artificial and still never solve the problem.   I need to do some learning
> here, coming from windows the file locking was pretty exclusive.  I want to
> see about FileChannel locks in nio and Linux file management.    This could
> maybe be an area to look at.  Right now there are no locks obtained for the
> file being processed.
>
> I will come back with something a little better formulated soon...
>
> Thanks
>
>
> Phil Scala
> Software Developer / Architect
> Global Relay
>
> [EMAIL PROTECTED]
>
> 866.484.6630  |  [EMAIL PROTECTED]  |  globalrelay.com
>
> -----Original Message-----
> From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]] On Behalf Of Edward
> Sargisson
> Sent: Wednesday, May 22, 2013 12:22 PM
> To: Mike Percy; [EMAIL PROTECTED]
> Subject: Re: spooldir source reading Flume itself and thinking the file
> has changed (1.3.1)
>
> Hi Mike,
> I haven't tried log4j2 in my environments but my review of the log4j2
> change is that it should work.
>
> What would I change?
> Phil Scala may have some thoughts.
>
> It would be nice if we thought through the file locking. I want to be able
> to put a file in the spooldir and know that Flume isn't going to get
> started until I'm ready. This certainly involves thinking about what the
> file-putting process is doing but it's not clear to me how to ensure this
> whole part is safe.
>
> The thing that is currently annoying is handling stack traces. All logging
> systems I've seen (except recent log4j2) output the stack trace with each
> frame on a new line. This means that each frame gets its own log event and
> the timestamp has to be added by Flume (instead of taken from the original
> event). That Flume timestamp might be delayed by up to 1 minute (because of
> log rolling so its pretty crap). Logstash has a multiline filter that
> somewhat solves this.
>
> My current approach is to try and get the Log4j2 FlumeAppender and Flume
> 1.3.1 reliable and trustworthy.
>
> Cheers,
> Edward
>
> "Hi Edward,
> Did the fixes in LOG4J2-254 fix your file rolling issue?
>
> What are your thoughts on how to improve spooling directory source's error
> handling when it detects a change in the file? Just bail and retry later? I
> suppose that's a pretty reasonable approach.
>
> Regards,
> Mike
>
>
> On Tue, May 14, 2013 at 4:50 PM, Edward Sargisson <[EMAIL PROTECTED]
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB