Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume, mail # dev - Review Request: FLUME-1947. ExecSource should optionally block if the channel is full.


Copy link to this message
-
Re: Review Request: FLUME-1947. ExecSource should optionally block if the channel is full.
Hari Shreedharan 2013-06-18, 20:05

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/9914/
-----------------------------------------------------------

(Updated June 18, 2013, 8:05 p.m.)
Review request for Flume.
Changes
-------

Reworked the patch to remove the flushing automatically changes since it has already been committed.
Description
-------

Made multiple changes to the exec source:

1. Flush the events to the channel, based on a configurable timeout - even when a full batch was not received.
2. Block the spawned process until channel exceptions are cleared up by not reading the stdout buffer. Backoff exponentially (subject to an upper limit), till events are available.
This addresses bug FLUME-1947.
    https://issues.apache.org/jira/browse/FLUME-1947
Diffs (updated)
-----

  flume-ng-core/src/main/java/org/apache/flume/source/ExecSource.java 1d8d267
  flume-ng-core/src/main/java/org/apache/flume/source/ExecSourceConfigurationConstants.java 957ec7f
  flume-ng-core/src/test/java/org/apache/flume/source/TestExecSource.java 54f71a1

Diff: https://reviews.apache.org/r/9914/diff/
Testing
-------

Added unit tests for the new features. All current tests pass.
Thanks,

Hari Shreedharan