Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume, mail # dev - Review Request: FLUME-2027: Check for default replication fails on federated cluster in hdfs sink


Copy link to this message
-
Re: Review Request: FLUME-2027: Check for default replication fails on federated cluster in hdfs sink
Mike Percy 2013-05-01, 07:36

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/10834/
-----------------------------------------------------------

(Updated May 1, 2013, 7:36 a.m.)
Review request for Flume.
Description
-------

Federated HDFS has a new API which has been ported to Hadoop 1.1.0 and Hadoop 2.0.0-alpha from HADOOP-8014 which takes a Path argument to the FileSystem.getDefaultReplication() call. This patch simply moves the existing implementation over to that new API.
This addresses bug FLUME-2027.
    https://issues.apache.org/jira/browse/FLUME-2027
Diffs
-----

  flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/AbstractHDFSWriter.java ff4f223
  flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSCompressedDataStream.java 0c618b5
  flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSDataStream.java c87fafe
  flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSSequenceFile.java 1a401d6

Diff: https://reviews.apache.org/r/10834/diff/
Testing (updated)
-------

Unit tests pass.

Tested this by running the unit tests under both Hadoop 1.0.1 and Hadoop 1.1.2 (manually modified the pom files) and looking at the DEBUG output from the unit tests. The debug output indicated that HADOOP-8014 is used if it's available, otherwise the existing API is used. This should be sufficient for all cases.
Thanks,

Mike Percy