Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # dev >> Review Request 13640: HDFS Sink rollSize is calculated based off of uncompressed size of cumulative events.


Copy link to this message
-
Re: Review Request 13640: HDFS Sink rollSize is calculated based off of uncompressed size of cumulative events.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/13640/#review27480
-----------------------------------------------------------

flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java
<https://reviews.apache.org/r/13640/#comment53399>

    I really think only one of the two should be accepted. Yes, you don't want to change the behavior of rollSize, but having this check for rollCompressedSize after rollSize makes it moot, since the uncompressed size of the events is almost always going to be > the compressed size. So in effect, the roll size is parameter will be processed first unless you set rollCompressedSize to be far less than rollSize. Most often users do want the rollCompressedSize behavior.
- Hari Shreedharan
On Sept. 19, 2013, 9:04 p.m., Ted Malaska wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/13640/
> -----------------------------------------------------------
>
> (Updated Sept. 19, 2013, 9:04 p.m.)
>
>
> Review request for Flume.
>
>
> Bugs: 2128
>     https://issues.apache.org/jira/browse/2128
>
>
> Repository: flume-git
>
>
> Description
> -------
>
> 1) Added support for hdfs.rollCompressedSize
> 2) Added unit test
> 3) Updated doc
>
>
> Diffs
> -----
>
>   flume-ng-doc/sphinx/FlumeUserGuide.rst bbfb5d0
>   flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/AbstractHDFSWriter.java da0466d
>   flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java 65f4d2c
>   flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSCompressedDataStream.java 5518547
>   flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java f0a6e4b
>   flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSSequenceFile.java 5fe9f1b
>   flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSWriter.java 44a984a
>   flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/MockHDFSWriter.java ec49b97
>   flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestBucketWriter.java f741e03
>
> Diff: https://reviews.apache.org/r/13640/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Ted Malaska
>
>