Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume >> mail # dev >> Review Request: RollingFileSink need to be able to construct directory path based on escape sequence


+
Ted Malaska 2012-09-17, 15:49
+
Brock Noland 2012-09-17, 17:58
+
Ted Malaska 2012-09-17, 18:06
+
Ted Malaska 2012-09-17, 22:34
Copy link to this message
-
Re: Review Request: RollingFileSink need to be able to construct directory path based on escape sequence

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/7131/#review14021
-----------------------------------------------------------
Echoing my comments from the JIRA. Personally what I think we should do with this patch is not modify HDFSEventSink but get all the "infrastructure" ready so we could modify HDFSEventSink once we are sure the new code is working correctly. The reason is HDFSEventSink is used by a ton of people and it's been a source of bugs in the past.
- Brock Noland
On Sept. 17, 2012, 10:34 p.m., Ted Malaska wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/7131/
> -----------------------------------------------------------
>
> (Updated Sept. 17, 2012, 10:34 p.m.)
>
>
> Review request for Flume.
>
>
> Description
> -------
>
> OK, first note this is still a work in progress, but I'm at the point where I need feedback.
>
> Initial goal was to allow RollingFileSink to do Escape Sequence the same way as HDFSEventSink.
>
> In order to reuse code, my change attempt to reuse bucketWriter and BucketWriterLinkedHashMap.
>
> BucketWriter has been broken up into three files.
>
> > AbstractBucketWriter : Contains all the rollover and escape sequence logic.
> > BucketWriter: Contains the IO logic to HDFS
> > FileBucketWriter: contains the IO logic to normal files
>
> I also added a couple unit tests to RollingFileSink for new types of roll overs and escape sequences.
>
>
> This addresses bug FLUME-1295.
>     https://issues.apache.org/jira/browse/FLUME-1295
>
>
> Diffs
> -----
>
>   flume-ng-core/src/main/java/org/apache/flume/sink/AbstractBucketWriter.java PRE-CREATION
>   flume-ng-core/src/main/java/org/apache/flume/sink/BucketWriterLinkedHashMap.java PRE-CREATION
>   flume-ng-core/src/main/java/org/apache/flume/sink/FileBucketWriter.java PRE-CREATION
>   flume-ng-core/src/main/java/org/apache/flume/sink/RollingFileSink.java be640bb
>   flume-ng-core/src/test/java/org/apache/flume/sink/TestRollingFileSink.java 07fa644
>   flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java 6408eb9
>   flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java 9a76ecb
>
> Diff: https://reviews.apache.org/r/7131/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Ted Malaska
>
>