Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # dev >> Review Request: Flume-1685: ExecSource shouldn't die if the channel is full


Copy link to this message
-
Review Request: Flume-1685: ExecSource shouldn't die if the channel is full

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/8987/
-----------------------------------------------------------

Review request for Flume.
Description
-------

Rather than kill the source outright when a channel fills, the channel should throw away the event (since there is nowhere to put it).  A basic back-off mechanism is also implemented so retries get spaced out the longer the channel remains full up to a configurable limit.

Patches are attached to FLUME-1685 including a backport to the 1.2 branch (which I use with Cloudera's cdh3u5 distribution)
This addresses bug FLUME-1685.
    https://issues.apache.org/jira/browse/FLUME-1685
Diffs
-----
Diff: https://reviews.apache.org/r/8987/diff/
Testing
-------

A unit test was created to test the full channel condition.
This patch has been running in our production environment for 2 months and has successfully recovered several times from full channel events.
Thanks,

Steve Hoffman