Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume, mail # dev - Review Request: FLUME-1649. Improve HBaseSink performance.


Copy link to this message
-
Re: Review Request: FLUME-1649. Improve HBaseSink performance.
Brock Noland 2012-12-07, 00:52


> On Dec. 6, 2012, 10:41 p.m., Brock Noland wrote:
> > flume-ng-sinks/flume-ng-hbase-sink/src/main/java/org/apache/flume/sink/hbase/HBaseSink.java, line 244
> > <https://reviews.apache.org/r/7657/diff/3/?file=177947#file177947line244>
> >
> >     Why do we batch the puts into one runnable but not the increments?
>
> Hari Shreedharan wrote:
>     This is because Increments did not implement Row interface in 0.92.1 release (which we use in the hadoop-1 profile), and so they cannot be batched

OK, perfect, I was not aware of this. Not being familiar with the HBase api, I still have one question, is there reason we don't send a list of Increments like we send a list of Rows in a single PutRunnable?
- Brock
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/7657/#review14116
-----------------------------------------------------------
On Oct. 19, 2012, 12:14 a.m., Hari Shreedharan wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/7657/
> -----------------------------------------------------------
>
> (Updated Oct. 19, 2012, 12:14 a.m.)
>
>
> Review request for Flume.
>
>
> Description
> -------
>
> Modify the Hbase sink to use multiple threads to wait on RPC calls. Since the threads are blocked often, this will not cause too many running threads.
>
>
> This addresses bug FLUME-1649.
>     https://issues.apache.org/jira/browse/FLUME-1649
>
>
> Diffs
> -----
>
>   flume-ng-sinks/flume-ng-hbase-sink/src/main/java/org/apache/flume/sink/hbase/HBaseSink.java 021ecd0
>
> Diff: https://reviews.apache.org/r/7657/diff/
>
>
> Testing
> -------
>
> All current unit tests pass. Did functional testing.
>
>
> Thanks,
>
> Hari Shreedharan
>
>