Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume, mail # dev - Review Request: Flume-1685: ExecSource shouldn't die if the channel is full


+
Steve Hoffman 2013-01-17, 02:49
+
Brock Noland 2013-01-17, 03:21
+
Hari Shreedharan 2013-01-17, 04:08
+
Hari Shreedharan 2013-01-17, 04:11
+
Steve Hoffman 2013-01-17, 14:19
+
Steve Hoffman 2013-01-17, 15:08
Copy link to this message
-
Re: Review Request: FLUME-1685: ExecSource shouldn't die if the channel is full
Steve Hoffman 2013-01-17, 15:12

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/9003/#review15454
-----------------------------------------------------------

.gitignore
<https://reviews.apache.org/r/9003/#comment33333>

    I added these other intellij files as ignores since you already included the iml files and we wouldn't want this committed accidentally.

flume-ng-core/pom.xml
<https://reviews.apache.org/r/9003/#comment33334>

    Added mocking framework for unit testing

flume-ng-core/src/main/java/org/apache/flume/source/ExecSource.java
<https://reviews.apache.org/r/9003/#comment33335>

    You don't override an interface so I removed this.  IntelliJ (correctly) complains.

flume-ng-core/src/main/java/org/apache/flume/source/ExecSource.java
<https://reviews.apache.org/r/9003/#comment33336>

    You don't override an interface so I removed this.  IntelliJ (correctly) complains.
- Steve Hoffman
On Jan. 17, 2013, 3:08 p.m., Steve Hoffman wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/9003/
> -----------------------------------------------------------
>
> (Updated Jan. 17, 2013, 3:08 p.m.)
>
>
> Review request for Flume.
>
>
> Description
> -------
>
> Rather than kill the source outright when a channel fills, the channel should throw away the event (since there is nowhere to put it).  A basic back-off mechanism is also implemented so retries get spaced out the longer the channel remains full up to a configurable limit.
>
> Patches are attached to FLUME-1685 including a backport to the 1.2 branch (which I use with Cloudera's cdh3u5 distribution)
>
>
> This addresses bug FLUME-1685.
>     https://issues.apache.org/jira/browse/FLUME-1685
>
>
> Diffs
> -----
>
>   .gitignore 08c33a0
>   flume-ng-core/pom.xml d01baca
>   flume-ng-core/src/main/java/org/apache/flume/source/ExecSource.java 46f672f
>   flume-ng-core/src/main/java/org/apache/flume/source/ExecSourceConfigurationConstants.java 0ba0508
>   flume-ng-core/src/test/java/org/apache/flume/source/TestExecSource.java 8bcf320
>
> Diff: https://reviews.apache.org/r/9003/diff/
>
>
> Testing
> -------
>
> A unit test was created to test the full channel condition.
> This patch has been running in our production environment for 2 months and has successfully recovered several times from full channel events.
>
>
> Thanks,
>
> Steve Hoffman
>
>

+
Mike Percy 2013-01-25, 23:19