Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # dev >> Re: Review Request 11583: HDFS Sink should check if file is closed and retry if it is not.


Copy link to this message
-
Re: Review Request 11583: HDFS Sink should check if file is closed and retry if it is not.


> On July 10, 2013, 12:24 a.m., Hari Shreedharan wrote:
> > flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/AbstractHDFSWriter.java, lines 70-72
> > <https://reviews.apache.org/r/11583/diff/4/?file=314913#file314913line70>
> >
> >     Hmm, if number of close retries is set to zero, then the timeBetweenCloseRetries should have no effect right? So we should "unset" the timeBetweenCloseRetries if numberOfCloseRetries < 0 - maybe set it to INT_MAX? But this really does not matter, since the closeHDFSOutputStream checks if we should retry.

I also made this change
> On July 10, 2013, 12:24 a.m., Hari Shreedharan wrote:
> > flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/AbstractHDFSWriter.java, line 253
> > <https://reviews.apache.org/r/11583/diff/4/?file=314913#file314913line253>
> >
> >     If we change this to number of close tries, then this should also take care of that, so we call close() method only that many times.
> >    
> >     To keep this code the same we should perhaps set numberOfCloseRetries to the value from the config - 1, but also check that the file is not closed before logging the error (to handle the case where the closeTries is set to 1)

This also effect line 62.
- Ted
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/11583/#review22940
-----------------------------------------------------------
On July 2, 2013, 1:18 p.m., Ted Malaska wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/11583/
> -----------------------------------------------------------
>
> (Updated July 2, 2013, 1:18 p.m.)
>
>
> Review request for Flume.
>
>
> Bugs: 2007
>     https://issues.apache.org/jira/browse/2007
>
>
> Repository: flume-git
>
>
> Description
> -------
>
> We can use the new API added in HDFS-4525. We will need to use reflection though, so we can run against a version of HDFS which does not have this API.
>
>
> Diffs
> -----
>
>   flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/AbstractHDFSWriter.java bc3b383
>   flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSCompressedDataStream.java 2c2be6a
>   flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSDataStream.java b8214be
>   flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSSequenceFile.java 0383744
>   flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/MockFileSystemCloseRetryWrapper.java PRE-CREATION
>   flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/MockFsDataOutputStreamCloseRetryWrapper.java PRE-CREATION
>   flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestUseRawLocalFileSystem.java ffbdde0
>
> Diff: https://reviews.apache.org/r/11583/diff/
>
>
> Testing
> -------
>
>
> File Attachments
> ----------------
>
> Patch-0
>   https://reviews.apache.org/media/uploaded/files/2013/05/31/FLUME-2007-0.patch
>
>
> Thanks,
>
> Ted Malaska
>
>