Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # dev >> Review Request: ExecSource don't flush the cache if there is no input entries


Copy link to this message
-
Re: Review Request: ExecSource don't flush the cache if there is no input entries

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/8854/#review15108
-----------------------------------------------------------

Ship it!
Ship It!

- Denny Ye
On Jan. 7, 2013, 5:57 a.m., Fengdong Yu wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/8854/
> -----------------------------------------------------------
>
> (Updated Jan. 7, 2013, 5:57 a.m.)
>
>
> Review request for Flume.
>
>
> Description
> -------
>
> ExecSource has a default batchSize: 20, exec source read data from the source, then put it into the cache, after the cache is full, push it to the channel.
>
> but if exec source's cache is not full, and there isn't any input for a long time, then these entries always kept in the cache, there is no chance to the channel until the source's cache is full.
>
> so, the patch added a new config line: batchTimeout for ExecSource, and default is 3 seconds, if batchTimeout exceeded, push all cached data to the channel even the cache is not full.
>
>
> Diffs
> -----
>
>   flume-ng-core/src/main/java/org/apache/flume/source/ExecSource.java 495b03f
>   flume-ng-core/src/main/java/org/apache/flume/source/ExecSourceConfigurationConstants.java 1b35b01
>
> Diff: https://reviews.apache.org/r/8854/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Fengdong Yu
>
>