Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # dev >> Re: Review Request 16415: flume-ng accumulo sink


Copy link to this message
-
Re: Review Request 16415: flume-ng accumulo sink


> On Jan. 9, 2014, 9:52 p.m., Hari Shreedharan wrote:
> > flume-ng-sinks/flume-ng-accumulo-sink/src/main/java/org/apache/flume/sink/accumulo/AccumuloSink.java, line 173
> > <https://reviews.apache.org/r/16415/diff/1/?file=401611#file401611line173>
> >
> >     Really, we should return backoff only if the batch is empty, not if we could not get a complete the batch
>
> Matt Molek wrote:
>     The way it's currently written, in the event of a null event, the partial batch is committed and then backoff is returned. That seems to be the way the HBaseSink works too (though see my question on an earlier issue about whether or not I should be following the way o.a.f.sink.hbase.HBaseSink does things).
>    
>     What is the desired behavior when a null event is received? And what exactly is an empty batch? Just a batch where the first event received is null?
>    
>     Thanks for the clarification.
>
> Hari Shreedharan wrote:
>     Yes, we should ideally back off only when an entire batch is null. So if the first event received in that batch is null, backoff - else don't. HBaseSink needs to be updated to do this, but all others already do. The HBaseSink is less commonly use compared to the AsyncHBaseSink, so many of the bug fixes and updates went into the AsyncHBaseSink.
>
> Matt Molek wrote:
>     Coming back to this so I can finish up and submit a new patch.
>    
>     There are two cases that need to be handled. First, an empty batch, which is when the first event in a batch is null. When that happens, close the transaction and return backoff.
>    
>     What should be done if the null event comes in the middle of a transaction? I'm having a little trouble understanding the ASyncHBaseSink code, but as far as I can tell, it takes events off the channel until hitting a null event, and then immediately breaks from taking more events, flushes the hbase client, commits the transaction, and returns backoff. (A lot of other stuff happens too, but I'm just trying to get down the the basic operations that are going on)
>    
>

Yes, on empty batch return backoff. In the middle of a transaction, if the channel returns null then it means there are no more events in the channel, so the transaction can be committed and success can be returned. You need to rollback the transaction only if the channel throws an exception before the transaction is committed (or the serializer throws)
- Hari
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/16415/#review31464
-----------------------------------------------------------
On Dec. 27, 2013, 6 p.m., Matt Molek wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/16415/
> -----------------------------------------------------------
>
> (Updated Dec. 27, 2013, 6 p.m.)
>
>
> Review request for Flume.
>
>
> Repository: flume-git
>
>
> Description
> -------
>
> This patch is identical to FLUME-2234-0.patch except I had to regenerate it with different git options to get review board to accept it.
>
>
> Diffs
> -----
>
>   flume-ng-sinks/flume-ng-accumulo-sink/pom.xml PRE-CREATION
>   flume-ng-sinks/flume-ng-accumulo-sink/src/main/java/org/apache/flume/sink/accumulo/AccumuloEventSerializer.java PRE-CREATION
>   flume-ng-sinks/flume-ng-accumulo-sink/src/main/java/org/apache/flume/sink/accumulo/AccumuloSink.java PRE-CREATION
>   flume-ng-sinks/flume-ng-accumulo-sink/src/main/java/org/apache/flume/sink/accumulo/AccumuloSinkConfigurationConstants.java PRE-CREATION
>   flume-ng-sinks/flume-ng-accumulo-sink/src/main/java/org/apache/flume/sink/accumulo/SimpleAccumuloEventSerializer.java PRE-CREATION
>   flume-ng-sinks/flume-ng-accumulo-sink/src/test/java/org/apache/flume/sink/accumulo/TestAccumuloSink.java PRE-CREATION
>   flume-ng-sinks/pom.xml d03576b
>
> Diff: https://reviews.apache.org/r/16415/diff/