Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # dev >> Review Request 16415: flume-ng accumulo sink


Copy link to this message
-
Re: Review Request 16415: flume-ng accumulo sink


> On Jan. 9, 2014, 9:52 p.m., Hari Shreedharan wrote:
> > flume-ng-sinks/flume-ng-accumulo-sink/src/main/java/org/apache/flume/sink/accumulo/AccumuloSink.java, line 173
> > <https://reviews.apache.org/r/16415/diff/1/?file=401611#file401611line173>
> >
> >     Really, we should return backoff only if the batch is empty, not if we could not get a complete the batch
>
> Matt Molek wrote:
>     The way it's currently written, in the event of a null event, the partial batch is committed and then backoff is returned. That seems to be the way the HBaseSink works too (though see my question on an earlier issue about whether or not I should be following the way o.a.f.sink.hbase.HBaseSink does things).
>    
>     What is the desired behavior when a null event is received? And what exactly is an empty batch? Just a batch where the first event received is null?
>    
>     Thanks for the clarification.

Yes, we should ideally back off only when an entire batch is null. So if the first event received in that batch is null, backoff - else don't. HBaseSink needs to be updated to do this, but all others already do. The HBaseSink is less commonly use compared to the AsyncHBaseSink, so many of the bug fixes and updates went into the AsyncHBaseSink.
- Hari
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/16415/#review31464
-----------------------------------------------------------
On Dec. 27, 2013, 6 p.m., Matt Molek wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/16415/
> -----------------------------------------------------------
>
> (Updated Dec. 27, 2013, 6 p.m.)
>
>
> Review request for Flume.
>
>
> Repository: flume-git
>
>
> Description
> -------
>
> This patch is identical to FLUME-2234-0.patch except I had to regenerate it with different git options to get review board to accept it.
>
>
> Diffs
> -----
>
>   flume-ng-sinks/flume-ng-accumulo-sink/pom.xml PRE-CREATION
>   flume-ng-sinks/flume-ng-accumulo-sink/src/main/java/org/apache/flume/sink/accumulo/AccumuloEventSerializer.java PRE-CREATION
>   flume-ng-sinks/flume-ng-accumulo-sink/src/main/java/org/apache/flume/sink/accumulo/AccumuloSink.java PRE-CREATION
>   flume-ng-sinks/flume-ng-accumulo-sink/src/main/java/org/apache/flume/sink/accumulo/AccumuloSinkConfigurationConstants.java PRE-CREATION
>   flume-ng-sinks/flume-ng-accumulo-sink/src/main/java/org/apache/flume/sink/accumulo/SimpleAccumuloEventSerializer.java PRE-CREATION
>   flume-ng-sinks/flume-ng-accumulo-sink/src/test/java/org/apache/flume/sink/accumulo/TestAccumuloSink.java PRE-CREATION
>   flume-ng-sinks/pom.xml d03576b
>
> Diff: https://reviews.apache.org/r/16415/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Matt Molek
>
>