Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # dev >> Review Request 14683: FLUME-2119 -- duplicate files cause flume to enter irrecoverable state


Copy link to this message
-
Review Request 14683: FLUME-2119 -- duplicate files cause flume to enter irrecoverable state

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14683/
-----------------------------------------------------------

Review request for Flume.
Bugs: FLUME-2119
    https://issues.apache.org/jira/browse/FLUME-2119
Repository: Flume
Description
-------

Added a new configuration setting "useStrictSpooledFilePolicies" to control when the ReliableSpoolingFileEventReader thorws and IllegalStateException.  This is so that if someone understands what they are doing a spooled file can be safely spooled more than 1 time.  This will also control cases where if the file being read is updated with a new modified date/time similarly an exception is not thrown.  

the default for this setting is "true" which means the code acts in the same way today, honoring the spool directory requirements as documented.  A user will have to add the setting and set it to false in order to have this take effect.

As mentioned in JIRA, Flume v1.5 does not spiral when a duplicate file is spooled, however it will "shutdown" requiring human intervention.
Diffs
-----
Diff: https://reviews.apache.org/r/14683/diff/
Testing
-------

4 new unit tests added, existing unit tests in TestReliableSpoolingFileEventReader not affected.  Manually executed some tests on a windows machine spooling the same named file (different contents) to ensure no side effects.
Thanks,

Phil Scala