Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume, mail # dev - Review Request: FLUME-2027: Check for default replication fails on federated cluster in hdfs sink


Copy link to this message
-
Re: Review Request: FLUME-2027: Check for default replication fails on federated cluster in hdfs sink
Hari Shreedharan 2013-05-06, 21:26

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/10834/#review20233
-----------------------------------------------------------

Ship it!
Ship It!

- Hari Shreedharan
On May 1, 2013, 7:36 a.m., Mike Percy wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/10834/
> -----------------------------------------------------------
>
> (Updated May 1, 2013, 7:36 a.m.)
>
>
> Review request for Flume.
>
>
> Description
> -------
>
> Federated HDFS has a new API which has been ported to Hadoop 1.1.0 and Hadoop 2.0.0-alpha from HADOOP-8014 which takes a Path argument to the FileSystem.getDefaultReplication() call. This patch simply moves the existing implementation over to that new API.
>
>
> This addresses bug FLUME-2027.
>     https://issues.apache.org/jira/browse/FLUME-2027
>
>
> Diffs
> -----
>
>   flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/AbstractHDFSWriter.java ff4f223
>   flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSCompressedDataStream.java 0c618b5
>   flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSDataStream.java c87fafe
>   flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSSequenceFile.java 1a401d6
>
> Diff: https://reviews.apache.org/r/10834/diff/
>
>
> Testing
> -------
>
> Unit tests pass.
>
> Tested this by running the unit tests under both Hadoop 1.0.1 and Hadoop 1.1.2 (manually modified the pom files) and looking at the DEBUG output from the unit tests. The debug output indicated that HADOOP-8014 is used if it's available, otherwise the existing API is used. This should be sufficient for all cases.
>
>
> Thanks,
>
> Mike Percy
>
>