Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume, mail # dev - Review Request: FLUME-1916: HDFS sink should poll for # of active replicas. If less than required, roll the file


+
Mike Percy 2013-02-28, 06:52
+
Hari Shreedharan 2013-02-28, 23:55
+
Mike Percy 2013-03-01, 01:55
+
Mike Percy 2013-03-01, 01:53
+
Hari Shreedharan 2013-03-01, 04:47
Copy link to this message
-
Re: Review Request: FLUME-1916: HDFS sink should poll for # of active replicas. If less than required, roll the file
Mike Percy 2013-03-01, 05:45


> On March 1, 2013, 4:47 a.m., Hari Shreedharan wrote:
> > flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java, line 430
> > <https://reviews.apache.org/r/9675/diff/2/?file=263764#file263764line430>
> >
> >     will not continue rolling files in this directory?

Fixed
> On March 1, 2013, 4:47 a.m., Hari Shreedharan wrote:
> > flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java, line 440
> > <https://reviews.apache.org/r/9675/diff/2/?file=263764#file263764line440>
> >
> >     Sorry, I think I missed this one earlier. At this point doRotate is always going to be true. Was this logic meant to check if shouldRotate() returned true because of under replication? If it did, don't rotate if it is under replicated? If it is, then you need to set doRotate to false in the if block above.

Wow, good catch! Fixed.
> On March 1, 2013, 4:47 a.m., Hari Shreedharan wrote:
> > flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java, line 484
> > <https://reviews.apache.org/r/9675/diff/2/?file=263764#file263764line484>
> >
> >     One more thing I missed here. If this a previous file owned by this bucket writer was under replicated, then isUnderReplicated is never reset to false, causing every following file to be rotated prematurely. It should either be done here or in the if(shouldRotate()) {..} block.

Added to open() call
- Mike
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/9675/#review17224
-----------------------------------------------------------
On March 1, 2013, 5:44 a.m., Mike Percy wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/9675/
> -----------------------------------------------------------
>
> (Updated March 1, 2013, 5:44 a.m.)
>
>
> Review request for Flume.
>
>
> Description
> -------
>
> This is based on https://issues.apache.org/jira/browse/HBASE-2234 ... essentially if datanodes are dying en masse then we want to close our files as we are writing so that we get new pipeline assignments from the namenode.
>
>
> This addresses bug FLUME-1916.
>     https://issues.apache.org/jira/browse/FLUME-1916
>
>
> Diffs
> -----
>
>   flume-ng-doc/sphinx/FlumeUserGuide.rst 5ac903e
>   flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/AbstractHDFSWriter.java PRE-CREATION
>   flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java cdc37f6
>   flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSCompressedDataStream.java 1f3521e
>   flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSDataStream.java 4ea4327
>   flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSSequenceFile.java 3bd25f4
>   flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSWriter.java abca21f
>   flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/MockHDFSWriter.java 5e8628b
>   flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestBucketWriter.java 99b6150
>   flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSinkOnMiniCluster.java 2e71069
>
> Diff: https://reviews.apache.org/r/9675/diff/
>
>
> Testing
> -------
>
> Added unit test.
>
>
> Thanks,
>
> Mike Percy
>
>

+
Mike Percy 2013-03-01, 05:44
+
Hari Shreedharan 2013-03-01, 05:58
+
Mike Percy 2013-03-01, 06:40
+
Mike Percy 2013-03-01, 06:38
+
Hari Shreedharan 2013-03-01, 08:58