Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # dev >> Review Request 20698: FLUME-2357. HDFS sink should retry closing files that previously had close errors


Copy link to this message
-
Re: Review Request 20698: FLUME-2357. HDFS sink should retry closing files that previously had close errors

This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/20698/

(Updated April 25, 2014, 5:11 a.m.)
Review request for Flume.
Changes

Added more tests. Removed some unused imports
Bugs: FLUME-2357
    https://issues.apache.org/jira/browse/FLUME-2357
Repository: flume-git
Description

Much of the size of the patch is due to a couple of file renames. Otherwise the patch itself is pretty simple. In the Bucketwriter, if a close fails, we simply reschedule the close to happen sometime later until it finally succeeds or till we hit a maximum count. Added a test case too. This depends on the presence of the isFileClosed method in the HDFS client API. If the method is absent, reattempts are not done.
Diffs (updated)

  flume-ng-doc/sphinx/FlumeUserGuide.rst 7b918ed
  flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/AbstractHDFSWriter.java da0466d
  flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java e82d13d
  flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSCompressedDataStream.java 5518547
  flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSDataStream.java e20d1ee
  flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java 4ea78c1
  flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSSequenceFile.java 5fe9f1b
  flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/MockDataStream.java PRE-CREATION
  flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/MockFileSystem.java PRE-CREATION
  flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/MockFileSystemCloseRetryWrapper.java b5d89e6
  flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/MockFsDataOutputStream.java PRE-CREATION
  flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/MockFsDataOutputStreamCloseRetryWrapper.java 1d8c140
  flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestBucketWriter.java b7cc586
  flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSink.java 87918d1
  flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestUseRawLocalFileSystem.java 4476530
  pom.xml 2aa0ad1

Diff: https://reviews.apache.org/r/20698/diff/
Testing

Added new unit test. All current tests pass.
Thanks,

Hari Shreedharan