-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/8987/
-----------------------------------------------------------

Review request for Flume.
Description
-------

Rather than kill the source outright when a channel fills, the channel should throw away the event (since there is nowhere to put it).  A basic back-off mechanism is also implemented so retries get spaced out the longer the channel remains full up to a configurable limit.

Patches are attached to FLUME-1685 including a backport to the 1.2 branch (which I use with Cloudera's cdh3u5 distribution)
This addresses bug FLUME-1685.
    https://issues.apache.org/jira/browse/FLUME-1685
Diffs
-----
Diff: https://reviews.apache.org/r/8987/diff/
Testing
-------

A unit test was created to test the full channel condition.
This patch has been running in our production environment for 2 months and has successfully recovered several times from full channel events.
Thanks,

Steve Hoffman

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/8987/#review15442
-----------------------------------------------------------
Something is odd, here. I only see two buttons on the right "Review" and "Ship it". Normally there is a "view diff" button.

- Brock Noland
On Jan. 17, 2013, 2:49 a.m., Steve Hoffman wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/8987/
> -----------------------------------------------------------
>
> (Updated Jan. 17, 2013, 2:49 a.m.)
>
>
> Review request for Flume.
>
>
> Description
> -------
>
> Rather than kill the source outright when a channel fills, the channel should throw away the event (since there is nowhere to put it).  A basic back-off mechanism is also implemented so retries get spaced out the longer the channel remains full up to a configurable limit.
>
> Patches are attached to FLUME-1685 including a backport to the 1.2 branch (which I use with Cloudera's cdh3u5 distribution)
>
>
> This addresses bug FLUME-1685.
>     https://issues.apache.org/jira/browse/FLUME-1685
>
>
> Diffs
> -----
>
>
> Diff: https://reviews.apache.org/r/8987/diff/
>
>
> Testing
> -------
>
> A unit test was created to test the full channel condition.
> This patch has been running in our production environment for 2 months and has successfully recovered several times from full channel events.
>
>
> Thanks,
>
> Steve Hoffman
>
>

> On Jan. 17, 2013, 3:21 a.m., Brock Noland wrote:
> > Something is odd, here. I only see two buttons on the right "Review" and "Ship it". Normally there is a "view diff" button.

Same here, though I can see those buttons on other patches.
- Hari
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/8987/#review15442
-----------------------------------------------------------
On Jan. 17, 2013, 2:49 a.m., Steve Hoffman wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/8987/
> -----------------------------------------------------------
>
> (Updated Jan. 17, 2013, 2:49 a.m.)
>
>
> Review request for Flume.
>
>
> Description
> -------
>
> Rather than kill the source outright when a channel fills, the channel should throw away the event (since there is nowhere to put it).  A basic back-off mechanism is also implemented so retries get spaced out the longer the channel remains full up to a configurable limit.
>
> Patches are attached to FLUME-1685 including a backport to the 1.2 branch (which I use with Cloudera's cdh3u5 distribution)
>
>
> This addresses bug FLUME-1685.
>     https://issues.apache.org/jira/browse/FLUME-1685
>
>
> Diffs
> -----
>
>
> Diff: https://reviews.apache.org/r/8987/diff/
>
>
> Testing
> -------
>
> A unit test was created to test the full channel condition.
> This patch has been running in our production environment for 2 months and has successfully recovered several times from full channel events.
>
>
> Thanks,
>
> Steve Hoffman
>
>

> On Jan. 17, 2013, 3:21 a.m., Brock Noland wrote:
> > Something is odd, here. I only see two buttons on the right "Review" and "Ship it". Normally there is a "view diff" button.
>
> Hari Shreedharan wrote:
>     Same here, though I can see those buttons on other patches.

Going here: https://reviews.apache.org/r/8987/diff/ gives a 404. Seems like there is no review associated with the request. The repo is set to "Flume" and not "flume-git."

Steve: Can you submit a new request with the repo as flume-git? That will allow you to upload your diff when you select the repo.
- Hari
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/8987/#review15442
-----------------------------------------------------------
On Jan. 17, 2013, 2:49 a.m., Steve Hoffman wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/8987/
> -----------------------------------------------------------
>
> (Updated Jan. 17, 2013, 2:49 a.m.)
>
>
> Review request for Flume.
>
>
> Description
> -------
>
> Rather than kill the source outright when a channel fills, the channel should throw away the event (since there is nowhere to put it).  A basic back-off mechanism is also implemented so retries get spaced out the longer the channel remains full up to a configurable limit.
>
> Patches are attached to FLUME-1685 including a backport to the 1.2 branch (which I use with Cloudera's cdh3u5 distribution)
>
>
> This addresses bug FLUME-1685.
>     https://issues.apache.org/jira/browse/FLUME-1685
>
>
> Diffs
> -----
>
>
> Diff: https://reviews.apache.org/r/8987/diff/
>
>
> Testing
> -------
>
> A unit test was created to test the full channel condition.
> This patch has been running in our production environment for 2 months and has successfully recovered several times from full channel events.
>
>
> Thanks,
>
> Steve Hoffman
>
>

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/9003/
-----------------------------------------------------------

Review request for Flume.
Description
-------

Rather than kill the source outright when a channel fills, the channel should throw away the event (since there is nowhere to put it).  A basic back-off mechanism is also implemented so retries get spaced out the longer the channel remains full up to a configurable limit.

Patches are attached to FLUME-1685 including a backport to the 1.2 branch (which I use with Cloudera's cdh3u5 distribution)
This addresses bug FLUME-1685.
    https://issues.apache.org/jira/browse/FLUME-1685
Diffs
-----

  .gitignore 08c33a0
  flume-ng-core/pom.xml d01baca
  flume-ng-core/src/main/java/org/apache/flume/source/ExecSource.java 46f672f
  flume-ng-core/src/main/java/org/apache/flume/source/ExecSourceConfigurationConstants.java 0ba0508
  flume-ng-core/src/test/java/org/apache/flume/source/TestExecSource.java 8bcf320

Diff: https://reviews.apache.org/r/9003/diff/
Testing
-------

A unit test was created to test the full channel condition.
This patch has been running in our production environment for 2 months and has successfully recovered several times from full channel events.
Thanks,

Steve Hoffman

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/9003/
-----------------------------------------------------------

(Updated Jan. 17, 2013, 3:08 p.m.)
Review request for Flume.
Changes
-------

Just capitalized the FLUME in the title
Summary (updated)
-----------------

FLUME-1685: ExecSource shouldn't die if the channel is full
Description
-------

Rather than kill the source outright when a channel fills, the channel should throw away the event (since there is nowhere to put it).  A basic back-off mechanism is also implemented so retries get spaced out the longer the channel remains full up to a configurable limit.

Patches are attached to FLUME-1685 including a backport to the 1.2 branch (which I use with Cloudera's cdh3u5 distribution)
This addresses bug FLUME-1685.
    https://issues.apache.org/jira/browse/FLUME-1685
Diffs
-----

  .gitignore 08c33a0
  flume-ng-core/pom.xml d01baca
  flume-ng-core/src/main/java/org/apache/flume/source/ExecSource.java 46f672f
  flume-ng-core/src/main/java/org/apache/flume/source/ExecSourceConfigurationConstants.java 0ba0508
  flume-ng-core/src/test/java/org/apache/flume/source/TestExecSource.java 8bcf320

Diff: https://reviews.apache.org/r/9003/diff/
Testing
-------

A unit test was created to test the full channel condition.
This patch has been running in our production environment for 2 months and has successfully recovered several times from full channel events.
Thanks,

Steve Hoffman

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/9003/#review15454
-----------------------------------------------------------

.gitignore
<https://reviews.apache.org/r/9003/#comment33333>

    I added these other intellij files as ignores since you already included the iml files and we wouldn't want this committed accidentally.

flume-ng-core/pom.xml
<https://reviews.apache.org/r/9003/#comment33334>

    Added mocking framework for unit testing

flume-ng-core/src/main/java/org/apache/flume/source/ExecSource.java
<https://reviews.apache.org/r/9003/#comment33335>

    You don't override an interface so I removed this.  IntelliJ (correctly) complains.

flume-ng-core/src/main/java/org/apache/flume/source/ExecSource.java
<https://reviews.apache.org/r/9003/#comment33336>

    You don't override an interface so I removed this.  IntelliJ (correctly) complains.
- Steve Hoffman
On Jan. 17, 2013, 3:08 p.m., Steve Hoffman wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/9003/
> -----------------------------------------------------------
>
> (Updated Jan. 17, 2013, 3:08 p.m.)
>
>
> Review request for Flume.
>
>
> Description
> -------
>
> Rather than kill the source outright when a channel fills, the channel should throw away the event (since there is nowhere to put it).  A basic back-off mechanism is also implemented so retries get spaced out the longer the channel remains full up to a configurable limit.
>
> Patches are attached to FLUME-1685 including a backport to the 1.2 branch (which I use with Cloudera's cdh3u5 distribution)
>
>
> This addresses bug FLUME-1685.
>     https://issues.apache.org/jira/browse/FLUME-1685
>
>
> Diffs
> -----
>
>   .gitignore 08c33a0
>   flume-ng-core/pom.xml d01baca
>   flume-ng-core/src/main/java/org/apache/flume/source/ExecSource.java 46f672f
>   flume-ng-core/src/main/java/org/apache/flume/source/ExecSourceConfigurationConstants.java 0ba0508
>   flume-ng-core/src/test/java/org/apache/flume/source/TestExecSource.java 8bcf320
>
> Diff: https://reviews.apache.org/r/9003/diff/
>
>
> Testing
> -------
>
> A unit test was created to test the full channel condition.
> This patch has been running in our production environment for 2 months and has successfully recovered several times from full channel events.
>
>
> Thanks,
>
> Steve Hoffman
>
>

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/9003/#review15720
-----------------------------------------------------------
Hi Steve,
I'm not a fan of dropping events. How about providing a configuration option here:
1. Default behavior: Upon ChannelException, hold the current Event batch in memory, back off exponentially (up to a configured MAX_BACKOFF), and retry until it succeeds. Do not read additional data from the InputStream until the put() succeeds.
2. Configuration option to drop when the channel fills up (i.e., this patch)

There is nothing in Flume today that knowingly throws away data by default.

- Mike Percy
On Jan. 17, 2013, 3:08 p.m., Steve Hoffman wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/9003/
> -----------------------------------------------------------
>
> (Updated Jan. 17, 2013, 3:08 p.m.)
>
>
> Review request for Flume.
>
>
> Description
> -------
>
> Rather than kill the source outright when a channel fills, the channel should throw away the event (since there is nowhere to put it).  A basic back-off mechanism is also implemented so retries get spaced out the longer the channel remains full up to a configurable limit.
>
> Patches are attached to FLUME-1685 including a backport to the 1.2 branch (which I use with Cloudera's cdh3u5 distribution)
>
>
> This addresses bug FLUME-1685.
>     https://issues.apache.org/jira/browse/FLUME-1685
>
>
> Diffs
> -----
>
>   .gitignore 08c33a0
>   flume-ng-core/pom.xml d01baca
>   flume-ng-core/src/main/java/org/apache/flume/source/ExecSource.java 46f672f
>   flume-ng-core/src/main/java/org/apache/flume/source/ExecSourceConfigurationConstants.java 0ba0508
>   flume-ng-core/src/test/java/org/apache/flume/source/TestExecSource.java 8bcf320
>
> Diff: https://reviews.apache.org/r/9003/diff/
>
>
> Testing
> -------
>
> A unit test was created to test the full channel condition.
> This patch has been running in our production environment for 2 months and has successfully recovered several times from full channel events.
>
>
> Thanks,
>
> Steve Hoffman
>
>