Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume, mail # dev - Review Request: Additional configuration parameters for HDFSSink


Copy link to this message
-
Review Request: Additional configuration parameters for HDFSSink
Thiruvalluvan M. G. 2013-04-26, 10:30

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/10606/
-----------------------------------------------------------

Review request for Flume.
Description
-------

This patch adds additional configuration parameters for HDFS Sink. They are for choosing the HDFS block size, HDFS replication factor and buffer size. These can now be chosen on per-sink basis.
This addresses bug FLUME-2003.
    https://issues.apache.org/jira/browse/FLUME-2003
Diffs
-----

  flume-ng-doc/sphinx/FlumeUserGuide.rst 693c0d7
  flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/AbstractHDFSWriter.java ff4f223
  flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSCompressedDataStream.java 0c618b5
  flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSDataStream.java c87fafe
  flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSSequenceFile.java 1a401d6

Diff: https://reviews.apache.org/r/10606/diff/
Testing
-------

I've tested this as follows:

(1) Without specifying these things work as before
(2) With these new parameters specified, the new HDFS files have the specified block size or replication or both.
Thanks,

Thiruvalluvan M. G.

+
Mike Percy 2013-05-08, 06:41