Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume, mail # dev - Review Request: Flume-1541: Implement a SinkSelector for LoadBalancingSinkProcessor that includes failover mechanics


Copy link to this message
-
Re: Review Request: Flume-1541: Implement a SinkSelector for LoadBalancingSinkProcessor that includes failover mechanics
Juhani Connolly 2012-09-06, 08:07


> On Sept. 6, 2012, 6:14 a.m., Mike Percy wrote:
> > Awesome... we definitely need exponential backoff on the load balancing sink processor.
> >
> > Only thing is, I wonder if we shouldn't just add it to the two existing selectors instead of implement new ones that essentially do the same thing. How about modifying the constructors to accept a boolean (backoffOnFailure) instead?
> >
> > Other than that it looks good, I saw only a couple minor edge case issues noted below.
>
> Mike Percy wrote:
>     Note: if we parameterize the constructors then of course we would add an additional config option to enable backoff on failure.

I find that while the code for both is quite the actual details are different enough that separating them made for cleaner more comprehensible code(this is not so much the case for the newly added random backoff though, where the code is very similar)

I wouldn't be strongly opposed if someone wanted to join them together, but personally prefer this.
- Juhani
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/6939/#review11083
-----------------------------------------------------------
On Sept. 6, 2012, 7:55 a.m., Juhani Connolly wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/6939/
> -----------------------------------------------------------
>
> (Updated Sept. 6, 2012, 7:55 a.m.)
>
>
> Review request for Flume.
>
>
> Description
> -------
>
> Added a callback to the SinkSelector interface to inform selectors of sink failures, adding a noop stub to the abstract source so existing selectors don't need changes.
>
> The new selector maintains counts of sequential failures as well as times for intended recovery and last failure time, which it uses to decide whether or not a sink should be added to the returned iterator. The iterator is generated in such a way that the round robin remains balanced.
>
>
> This addresses bug FLUME-1541.
>     https://issues.apache.org/jira/browse/FLUME-1541
>
>
> Diffs
> -----
>
>   flume-ng-core/src/main/java/org/apache/flume/sink/AbstractSinkSelector.java 63397a5
>   flume-ng-core/src/main/java/org/apache/flume/sink/LoadBalancingSinkProcessor.java 18d4509
>   flume-ng-core/src/test/java/org/apache/flume/sink/TestLoadBalancingSinkProcessor.java 1e9c94e
>   flume-ng-doc/sphinx/FlumeUserGuide.rst ffed72b
>
> Diff: https://reviews.apache.org/r/6939/diff/
>
>
> Testing
> -------
>
> Added 3 new tests to verify that dynamic rebalancing is done, that the sink becomes available again after timeout, and that the timeout increases with sequential failures
>
> All tests pass
>
>
> Thanks,
>
> Juhani Connolly
>
>