Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume, mail # dev - Review Request: FLUME-1916: HDFS sink should poll for # of active replicas. If less than required, roll the file


Copy link to this message
-
Re: Review Request: FLUME-1916: HDFS sink should poll for # of active replicas. If less than required, roll the file
Mike Percy 2013-03-01, 06:40


> On March 1, 2013, 5:58 a.m., Hari Shreedharan wrote:
> > flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java, lines 429-446
> > <https://reviews.apache.org/r/9675/diff/3/?file=263788#file263788line429>
> >
> >     I get that since consecutiveUnderReplCount is reset the counter gets reset on open, but you need to also set isUnderReplicated to false, otherwise once that is set to true, it is never reset to false and the consecutiveUnderReplCount can get incremented for every append following one rotation due to under replication.

Ugh, you are right. I tried to add a unit test for bringing back the DN but I couldn't get the NN to recognize it, so here is a fix without a unit test. I will try to add a working unit test for this condition in a follow up patch.
- Mike
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/9675/#review17229
-----------------------------------------------------------
On March 1, 2013, 6:38 a.m., Mike Percy wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/9675/
> -----------------------------------------------------------
>
> (Updated March 1, 2013, 6:38 a.m.)
>
>
> Review request for Flume.
>
>
> Description
> -------
>
> This is based on https://issues.apache.org/jira/browse/HBASE-2234 ... essentially if datanodes are dying en masse then we want to close our files as we are writing so that we get new pipeline assignments from the namenode.
>
>
> This addresses bug FLUME-1916.
>     https://issues.apache.org/jira/browse/FLUME-1916
>
>
> Diffs
> -----
>
>   flume-ng-doc/sphinx/FlumeUserGuide.rst 5ac903e
>   flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/AbstractHDFSWriter.java PRE-CREATION
>   flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java cdc37f6
>   flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSCompressedDataStream.java 1f3521e
>   flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSDataStream.java 4ea4327
>   flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSSequenceFile.java 3bd25f4
>   flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSWriter.java abca21f
>   flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/MockHDFSWriter.java 5e8628b
>   flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestBucketWriter.java 99b6150
>   flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSinkOnMiniCluster.java 2e71069
>
> Diff: https://reviews.apache.org/r/9675/diff/
>
>
> Testing
> -------
>
> Added unit test.
>
>
> Thanks,
>
> Mike Percy
>
>