Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume, mail # dev - Review Request 13463: FLUME-2157. Spool directory source does not shut down correctly when Flume is reconfigured.


+
Mike Percy 2013-08-10, 02:33
+
Roshan Naik 2013-08-12, 18:18
Copy link to this message
-
Re: Review Request 13463: FLUME-2157. Spool directory source does not shut down correctly when Flume is reconfigured.
Roshan Naik 2013-08-12, 18:19


> On Aug. 12, 2013, 6:18 p.m., Roshan Naik wrote:
> > In SpoolDirectorySource.shitdown(), it might be useful to add  INFO or DEBUG level log indicating that it is waiting  ...  before entering a potentially 10 second long wait for the executor to wind up.
> >
> > Looks good overall.

of course i meant SpoolDirectorySource.shutdown()   :-)
- Roshan
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/13463/#review25018
-----------------------------------------------------------
On Aug. 10, 2013, 2:33 a.m., Mike Percy wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/13463/
> -----------------------------------------------------------
>
> (Updated Aug. 10, 2013, 2:33 a.m.)
>
>
> Review request for Flume.
>
>
> Bugs: FLUME-2157
>     https://issues.apache.org/jira/browse/FLUME-2157
>
>
> Repository: flume-git
>
>
> Description
> -------
>
> Spool directory source does not shut down correctly when Flume is reconfigured. This can lead to all sorts of problems stemming from the fact that we have threads running which should not be. In the logs, it looks like multiple spool directory sources are running on the same directory, even though the config may say otherwise.
>
>
> Diffs
> -----
>
>   flume-ng-core/src/main/java/org/apache/flume/client/avro/ReliableSpoolingFileEventReader.java f82fe1f
>   flume-ng-core/src/main/java/org/apache/flume/source/SpoolDirectorySource.java 7145580
>   flume-ng-core/src/test/java/org/apache/flume/source/TestSpoolDirectorySource.java 652d2a2
>   flume-ng-tests/src/test/java/org/apache/flume/test/agent/TestSpooldirSource.java PRE-CREATION
>   flume-ng-tests/src/test/java/org/apache/flume/test/util/StagedInstall.java bc58340
>
> Diff: https://reviews.apache.org/r/13463/diff/
>
>
> Testing
> -------
>
> Added a unit test for this specific case and also an integration test that simply runs an agent with many concurrent spool directory sources.
>
>
> Thanks,
>
> Mike Percy
>
>