Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume >> mail # dev >> Review Request 13640: HDFS Sink rollSize is calculated based off of uncompressed size of cumulative events.


+
Ted Malaska 2013-08-19, 01:20
+
Jeff jlord 2013-09-18, 13:55
+
Hari Shreedharan 2013-09-18, 19:45
+
Ted Malaska 2013-09-19, 21:04
+
Hari Shreedharan 2013-10-24, 19:47
Copy link to this message
-
Re: Review Request 13640: HDFS Sink rollSize is calculated based off of uncompressed size of cumulative events.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/13640/#review27502
-----------------------------------------------------------
I have a few comments inline here and a bit more discussion on the JIRA itself.
flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/AbstractHDFSWriter.java
<https://reviews.apache.org/r/13640/#comment53426>

    This should be removed and implemented consistently.

flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSCompressedDataStream.java
<https://reviews.apache.org/r/13640/#comment53427>

    This is a potentially big lie... the compression buffer tends to hold a lot of data not yet flushed to the FS. See for example the gzip codec implementation.
- Mike Percy
On Sept. 19, 2013, 9:04 p.m., Ted Malaska wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/13640/
> -----------------------------------------------------------
>
> (Updated Sept. 19, 2013, 9:04 p.m.)
>
>
> Review request for Flume.
>
>
> Bugs: 2128
>     https://issues.apache.org/jira/browse/2128
>
>
> Repository: flume-git
>
>
> Description
> -------
>
> 1) Added support for hdfs.rollCompressedSize
> 2) Added unit test
> 3) Updated doc
>
>
> Diffs
> -----
>
>   flume-ng-doc/sphinx/FlumeUserGuide.rst bbfb5d0
>   flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/AbstractHDFSWriter.java da0466d
>   flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java 65f4d2c
>   flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSCompressedDataStream.java 5518547
>   flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java f0a6e4b
>   flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSSequenceFile.java 5fe9f1b
>   flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSWriter.java 44a984a
>   flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/MockHDFSWriter.java ec49b97
>   flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestBucketWriter.java f741e03
>
> Diff: https://reviews.apache.org/r/13640/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Ted Malaska
>
>

+
Ted Malaska 2013-11-05, 04:54