Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume, mail # dev - Review Request 14683: FLUME-2119 -- duplicate files cause flume to enter irrecoverable state


Copy link to this message
-
Re: Review Request 14683: FLUME-2119 -- duplicate files cause flume to enter irrecoverable state
Phil Scala 2013-10-16, 21:15

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14683/#review27087
-----------------------------------------------------------
- Phil Scala
On Oct. 16, 2013, 9:14 p.m., Phil Scala wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/14683/
> -----------------------------------------------------------
>
> (Updated Oct. 16, 2013, 9:14 p.m.)
>
>
> Review request for Flume.
>
>
> Bugs: FLUME-2119
>     https://issues.apache.org/jira/browse/FLUME-2119
>
>
> Repository: Flume
>
>
> Description
> -------
>
> Added a new configuration setting "useStrictSpooledFilePolicies" to control when the ReliableSpoolingFileEventReader thorws and IllegalStateException.  This is so that if someone understands what they are doing a spooled file can be safely spooled more than 1 time.  This will also control cases where if the file being read is updated with a new modified date/time similarly an exception is not thrown.  
>
> the default for this setting is "true" which means the code acts in the same way today, honoring the spool directory requirements as documented.  A user will have to add the setting and set it to false in order to have this take effect.
>
> As mentioned in JIRA, Flume v1.5 does not spiral when a duplicate file is spooled, however it will "shutdown" requiring human intervention.
>
>
> Diffs
> -----
>
>
> Diff: https://reviews.apache.org/r/14683/diff/
>
>
> Testing
> -------
>
> 4 new unit tests added, existing unit tests in TestReliableSpoolingFileEventReader not affected.  Manually executed some tests on a windows machine spooling the same named file (different contents) to ensure no side effects.
>
>
> File Attachments
> ----------------
>
> patch diff
>   https://reviews.apache.org/media/uploaded/files/2013/10/16/FLUME-2119-0.patch
>
>
> Thanks,
>
> Phil Scala
>
>