Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume, mail # dev - Review Request 13640: HDFS Sink rollSize is calculated based off of uncompressed size of cumulative events.


+
Ted Malaska 2013-08-19, 01:20
+
Jeff jlord 2013-09-18, 13:55
+
Hari Shreedharan 2013-09-18, 19:45
Copy link to this message
-
Re: Review Request 13640: HDFS Sink rollSize is calculated based off of uncompressed size of cumulative events.
Ted Malaska 2013-09-19, 21:04

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/13640/
-----------------------------------------------------------

(Updated Sept. 19, 2013, 9:04 p.m.)
Review request for Flume.
Changes
-------

Added to HDFSCompressedDataStream

  public long getLength() throws IOException {
    if (fsOut != null) {
      return fsOut.getPos();
    }
    return -1;
  }
Bugs: 2128
    https://issues.apache.org/jira/browse/2128
Repository: flume-git
Description
-------

1) Added support for hdfs.rollCompressedSize
2) Added unit test
3) Updated doc
Diffs (updated)
-----

  flume-ng-doc/sphinx/FlumeUserGuide.rst bbfb5d0
  flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/AbstractHDFSWriter.java da0466d
  flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java 65f4d2c
  flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSCompressedDataStream.java 5518547
  flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java f0a6e4b
  flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSSequenceFile.java 5fe9f1b
  flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSWriter.java 44a984a
  flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/MockHDFSWriter.java ec49b97
  flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestBucketWriter.java f741e03

Diff: https://reviews.apache.org/r/13640/diff/
Testing
-------
Thanks,

Ted Malaska

+
Hari Shreedharan 2013-10-24, 19:47
+
Mike Percy 2013-10-25, 01:35
+
Ted Malaska 2013-11-05, 04:54