Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume >> mail # dev >> Review Request: FLUME-1660 Close "idle" hdfs handles


+
Juhani Connolly 2012-10-19, 04:31
+
Juhani Connolly 2012-10-19, 06:01
+
Mike Percy 2012-10-29, 23:49
+
Mike Percy 2012-10-29, 23:51
+
Juhani Connolly 2012-10-31, 03:56
+
Juhani Connolly 2012-10-31, 06:11
+
Juhani Connolly 2012-10-31, 06:12
+
Mike Percy 2012-10-31, 07:27
+
Mike Percy 2012-10-31, 07:44
+
Juhani Connolly 2012-10-31, 07:46
+
Juhani Connolly 2012-10-31, 07:48
+
Juhani Connolly 2012-10-31, 08:19
+
Juhani Connolly 2012-10-31, 10:49
+
Juhani Connolly 2012-10-31, 10:53
+
Mike Percy 2012-11-02, 09:34
+
Alexander Alten-Lorenz 2012-11-06, 08:36
+
Juhani Connolly 2012-11-07, 01:34
+
Alexander Alten-Lorenz 2012-11-07, 09:15
+
Mike Percy 2012-11-09, 20:01
Copy link to this message
-
Re: Review Request: FLUME-1660 Close "idle" hdfs handles


> On Nov. 9, 2012, 8:01 p.m., Mike Percy wrote:
> > This looks good, but I don't see any unit tests that exercise the functionality.

Thats true, missed this. Was patching my local branch with and let run the build and a simple test. Sry.

- Ship it, please add a valuable unit test
- Alexander
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/7659/#review13301
-----------------------------------------------------------
On Nov. 7, 2012, 1:34 a.m., Juhani Connolly wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/7659/
> -----------------------------------------------------------
>
> (Updated Nov. 7, 2012, 1:34 a.m.)
>
>
> Review request for Flume and Mike Percy.
>
>
> Description
> -------
>
> Added lastWrite to BucketWriter to verify when it was last updated
>
> Added a thread to HDFSEventSink which verifies the last update of each active bucketWriter and closes them after the configurable timeout hdfs.closeIdleTimeout has passed.
>
>
> This addresses bug FLUME-1660.
>     https://issues.apache.org/jira/browse/FLUME-1660
>
>
> Diffs
> -----
>
>   flume-ng-doc/sphinx/FlumeUserGuide.rst c1303e0
>   flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java 9f2c763
>   flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java e369604
>   flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestBucketWriter.java 6a8072e
>
> Diff: https://reviews.apache.org/r/7659/diff/
>
>
> Testing
> -------
>
> Local machine testing was performed and the correct closing of files was confirmed, as well as the correct behavior of the configuration setting including disabling the feature(by using the default value for hdfs.closeIdleTimeout of 0)
>
>
> There is one unrelated test failure which I'm not sure of(if anyone knows what's causing this, please let me know)
>
> Failed tests:   testInOut(org.apache.flume.test.agent.TestFileChannel): Expected FILE_ROLL sink's dir to have only 1 child, but found 0 children. expected:<1> but was:<0>
>
> All other tests pass.
>
>
> Thanks,
>
> Juhani Connolly
>
>

+
Juhani Connolly 2012-11-12, 03:07
+
Juhani Connolly 2012-11-14, 08:01
+
Juhani Connolly 2012-11-16, 02:00
+
Mike Percy 2012-11-16, 07:25
+
Mike Percy 2012-11-16, 07:27
+
Juhani Connolly 2012-11-16, 08:10
+
Mike Percy 2012-11-16, 07:33
+
Juhani Connolly 2012-11-16, 08:12
+
Juhani Connolly 2012-11-16, 10:06
+
Mike Percy 2012-11-19, 08:17
+
Mike Percy 2012-11-19, 08:53