Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume, mail # dev - Review Request: FLUME-1916: HDFS sink should poll for # of active replicas. If less than required, roll the file


Copy link to this message
-
Re: Review Request: FLUME-1916: HDFS sink should poll for # of active replicas. If less than required, roll the file
Mike Percy 2013-03-01, 01:53

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/9675/
-----------------------------------------------------------

(Updated March 1, 2013, 1:53 a.m.)
Review request for Flume.
Changes
-------

Rebase & feedback from Hari
Description
-------

This is based on https://issues.apache.org/jira/browse/HBASE-2234 ... essentially if datanodes are dying en masse then we want to close our files as we are writing so that we get new pipeline assignments from the namenode.
This addresses bug FLUME-1916.
    https://issues.apache.org/jira/browse/FLUME-1916
Diffs (updated)
-----

  flume-ng-doc/sphinx/FlumeUserGuide.rst 5ac903e
  flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/AbstractHDFSWriter.java PRE-CREATION
  flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java cdc37f6
  flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSCompressedDataStream.java 1f3521e
  flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSDataStream.java 4ea4327
  flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSSequenceFile.java 3bd25f4
  flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSWriter.java abca21f
  flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/MockHDFSWriter.java 5e8628b
  flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestBucketWriter.java 99b6150
  flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSinkOnMiniCluster.java 2e71069

Diff: https://reviews.apache.org/r/9675/diff/
Testing
-------

Added unit test.
Thanks,

Mike Percy