Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume, mail # dev - Review Request: FLUME-1916: HDFS sink should poll for # of active replicas. If less than required, roll the file


+
Mike Percy 2013-02-28, 06:52
Copy link to this message
-
Re: Review Request: FLUME-1916: HDFS sink should poll for # of active replicas. If less than required, roll the file
Hari Shreedharan 2013-02-28, 23:55

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/9675/#review17213
-----------------------------------------------------------
+1. Looks good. I have a couple of comments which are nitpicks. If you think they need not be fixed, I will commit this patch.
flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/AbstractHDFSWriter.java
<https://reviews.apache.org/r/9675/#comment36504>

    nit: This getFsDesiredReplication call can be avoided if configuredMinReplicas != null by moving that call to an else block.

flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/AbstractHDFSWriter.java
<https://reviews.apache.org/r/9675/#comment36507>

    Is this method really required? If fs is null it means the registerCurrentStream was not called (or the unregister method was called). Can fs become null in any other case?
- Hari Shreedharan
On Feb. 28, 2013, 6:52 a.m., Mike Percy wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/9675/
> -----------------------------------------------------------
>
> (Updated Feb. 28, 2013, 6:52 a.m.)
>
>
> Review request for Flume.
>
>
> Description
> -------
>
> This is based on https://issues.apache.org/jira/browse/HBASE-2234 ... essentially if datanodes are dying en masse then we want to close our files as we are writing so that we get new pipeline assignments from the namenode.
>
>
> This addresses bug FLUME-1916.
>     https://issues.apache.org/jira/browse/FLUME-1916
>
>
> Diffs
> -----
>
>   flume-ng-doc/sphinx/FlumeUserGuide.rst 5ac903e
>   flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/AbstractHDFSWriter.java PRE-CREATION
>   flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java 0786857
>   flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSCompressedDataStream.java 1f3521e
>   flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSDataStream.java 4ea4327
>   flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSSequenceFile.java 3bd25f4
>   flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSWriter.java abca21f
>   flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/MockHDFSWriter.java 5e8628b
>   flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestBucketWriter.java ebe277c
>   flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSinkOnMiniCluster.java 2e71069
>
> Diff: https://reviews.apache.org/r/9675/diff/
>
>
> Testing
> -------
>
> Added unit test.
>
>
> Thanks,
>
> Mike Percy
>
>

+
Mike Percy 2013-03-01, 01:55
+
Mike Percy 2013-03-01, 01:53
+
Hari Shreedharan 2013-03-01, 04:47
+
Mike Percy 2013-03-01, 05:45
+
Mike Percy 2013-03-01, 05:44
+
Hari Shreedharan 2013-03-01, 05:58
+
Mike Percy 2013-03-01, 06:40
+
Mike Percy 2013-03-01, 06:38
+
Hari Shreedharan 2013-03-01, 08:58