Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume, mail # dev - Review Request: Flume-1541: Implement a SinkSelector for LoadBalancingSinkProcessor that includes failover mechanics


Copy link to this message
-
Re: Review Request: Flume-1541: Implement a SinkSelector for LoadBalancingSinkProcessor that includes failover mechanics
Juhani Connolly 2012-09-06, 07:55

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/6939/
-----------------------------------------------------------

(Updated Sept. 6, 2012, 7:55 a.m.)
Review request for Flume.
Changes
-------

Fixed the shift rollover

Abstracted out the backoff mechanism to avoid code repetition between roundrobin and random.
Added RANDOM_BACKOFF selector along with test for it(because of its random nature there is a negligible chance(something like 1/(2^49) that it fails if no random events are sent to the channel to be backed off).

I retained the old implementations. If someone else wants merge the plain and backoff versions and switch modes between them with a flag, they're welcome to... I'd prefer to avoid the confusion.

Added documents
Description
-------

Added a callback to the SinkSelector interface to inform selectors of sink failures, adding a noop stub to the abstract source so existing selectors don't need changes.

The new selector maintains counts of sequential failures as well as times for intended recovery and last failure time, which it uses to decide whether or not a sink should be added to the returned iterator. The iterator is generated in such a way that the round robin remains balanced.
This addresses bug FLUME-1541.
    https://issues.apache.org/jira/browse/FLUME-1541
Diffs (updated)
-----

  flume-ng-core/src/main/java/org/apache/flume/sink/AbstractSinkSelector.java 63397a5
  flume-ng-core/src/main/java/org/apache/flume/sink/LoadBalancingSinkProcessor.java 18d4509
  flume-ng-core/src/test/java/org/apache/flume/sink/TestLoadBalancingSinkProcessor.java 1e9c94e
  flume-ng-doc/sphinx/FlumeUserGuide.rst ffed72b

Diff: https://reviews.apache.org/r/6939/diff/
Testing
-------

Added 3 new tests to verify that dynamic rebalancing is done, that the sink becomes available again after timeout, and that the timeout increases with sequential failures

All tests pass
Thanks,

Juhani Connolly