Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume >> mail # dev >> Review Request 18668: Support coalescing increments in HBaseSink


+
Mike Percy 2014-03-02, 02:47
+
Mike Percy 2014-03-02, 13:10
Copy link to this message
-
Re: Review Request 18668: Support coalescing increments in HBaseSink

This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18668/

(Updated March 3, 2014, 5:09 a.m.)
Review request for Flume.
Changes

Fixed more HBase 0.96 issues
Bugs: FLUME-2338
    https://issues.apache.org/jira/browse/FLUME-2338
Repository: flume-git
Description

Coalesce support for HBaseSink. Also includes a BatchAware interface, allowing serializers that implement it to be aware of batching so that they can choose keys to optimize the effect of coalescing, if desired.
Diffs (updated)

  flume-ng-doc/sphinx/FlumeUserGuide.rst 96bf73e
  flume-ng-sinks/flume-ng-hbase-sink/src/main/java/org/apache/flume/sink/hbase/BatchAware.java PRE-CREATION
  flume-ng-sinks/flume-ng-hbase-sink/src/main/java/org/apache/flume/sink/hbase/HBaseSink.java c4a666c
  flume-ng-sinks/flume-ng-hbase-sink/src/test/java/org/apache/flume/sink/hbase/IncrementHBaseSerializer.java PRE-CREATION
  flume-ng-sinks/flume-ng-hbase-sink/src/test/java/org/apache/flume/sink/hbase/TestHBaseSink.java d1b0182

Diff: https://reviews.apache.org/r/18668/diff/
Testing

Added unit tests. Tested a version of this patch (prior to a rebase for trunk) under high load and the approach is very effective for certain high-increment workloads that end up with a lot of contention on keys in HBase.
Thanks,

Mike Percy