Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume, mail # dev - Review Request: RollingFileSink need to be able to construct directory path based on escape sequence


Copy link to this message
-
Review Request: RollingFileSink need to be able to construct directory path based on escape sequence
Ted Malaska 2012-09-17, 15:49

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/7131/
-----------------------------------------------------------

Review request for Flume.
Description
-------

OK, first note this is still a work in progress, but I'm at the point where I need feedback.

Initial goal was to allow RollingFileSink to do Escape Sequence the same way as HDFSEventSink.

In order to reuse code, my change attempt to reuse bucketWriter and BucketWriterLinkedHashMap.

BucketWriter has been broken up into three files.

> AbstractBucketWriter : Contains all the rollover and escape sequence logic.
> BucketWriter: Contains the IO logic to HDFS
> FileBucketWriter: contains the IO logic to normal files

I also added a couple unit tests to RollingFileSink for new types of roll overs and escape sequences.
This addresses bug FLUME-1295.
    https://issues.apache.org/jira/browse/FLUME-1295
Diffs
-----

  flume-ng-core/src/main/java/org/apache/flume/sink/AbstractBucketWriter.java PRE-CREATION
  flume-ng-core/src/main/java/org/apache/flume/sink/BucketWriterLinkedHashMap.java PRE-CREATION
  flume-ng-core/src/main/java/org/apache/flume/sink/FileBucketWriter.java PRE-CREATION
  flume-ng-core/src/main/java/org/apache/flume/sink/RollingFileSink.java be640bb
  flume-ng-core/src/test/java/org/apache/flume/sink/TestRollingFileSink.java 07fa644
  flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java 6408eb9
  flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java 9a76ecb

Diff: https://reviews.apache.org/r/7131/diff/
Testing
-------
Thanks,

Ted Malaska

+
Brock Noland 2012-09-17, 17:58
+
Ted Malaska 2012-09-17, 18:06
+
Ted Malaska 2012-09-17, 22:34
+
Brock Noland 2012-12-04, 16:37