Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume >> mail # dev >> Review Request: Flume-1685: ExecSource shouldn't die if the channel is full

Steve Hoffman 2013-01-17, 02:49
Brock Noland 2013-01-17, 03:21
Hari Shreedharan 2013-01-17, 04:08
Hari Shreedharan 2013-01-17, 04:11
Steve Hoffman 2013-01-17, 14:19
Copy link to this message
Re: Review Request: FLUME-1685: ExecSource shouldn't die if the channel is full

This is an automatically generated e-mail. To reply, visit:

(Updated Jan. 17, 2013, 3:08 p.m.)
Review request for Flume.

Just capitalized the FLUME in the title
Summary (updated)

FLUME-1685: ExecSource shouldn't die if the channel is full

Rather than kill the source outright when a channel fills, the channel should throw away the event (since there is nowhere to put it).  A basic back-off mechanism is also implemented so retries get spaced out the longer the channel remains full up to a configurable limit.

Patches are attached to FLUME-1685 including a backport to the 1.2 branch (which I use with Cloudera's cdh3u5 distribution)
This addresses bug FLUME-1685.

  .gitignore 08c33a0
  flume-ng-core/pom.xml d01baca
  flume-ng-core/src/main/java/org/apache/flume/source/ExecSource.java 46f672f
  flume-ng-core/src/main/java/org/apache/flume/source/ExecSourceConfigurationConstants.java 0ba0508
  flume-ng-core/src/test/java/org/apache/flume/source/TestExecSource.java 8bcf320

Diff: https://reviews.apache.org/r/9003/diff/

A unit test was created to test the full channel condition.
This patch has been running in our production environment for 2 months and has successfully recovered several times from full channel events.

Steve Hoffman

Steve Hoffman 2013-01-17, 15:12
Mike Percy 2013-01-25, 23:19