Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume, mail # dev - Re: Review Request 12060: Revised design for Spillable Mem Channel


Copy link to this message
-
Re: Review Request 12060: Revised design for Spillable Mem Channel
Hari Shreedharan 2013-10-31, 19:32

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/12060/#review27941
-----------------------------------------------------------
Sorry, I was looking at the code once again and found a couple issues which I think should be fixed.
flume-ng-channels/flume-spillable-memory-channel/src/main/java/org/apache/flume/channel/SpillableMemoryChannel.java
<https://reviews.apache.org/r/12060/#comment54379>

    If one of these two throws an exception, then the totalStored semaphore has the wrong value. You need to enclose the whole thing in a try-finally and release the totalStored if it was not successful. For the file channel, it is possible that a txn.take() throws, since it gets written to disk.

flume-ng-channels/flume-spillable-memory-channel/src/main/java/org/apache/flume/channel/SpillableMemoryChannel.java
<https://reviews.apache.org/r/12060/#comment54383>

    else should be on this line

flume-ng-channels/flume-spillable-memory-channel/src/main/java/org/apache/flume/channel/SpillableMemoryChannel.java
<https://reviews.apache.org/r/12060/#comment54391>

    Why is this not being done when the overflow channel is in use?
- Hari Shreedharan
On Oct. 24, 2013, 10:18 p.m., Roshan Naik wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/12060/
> -----------------------------------------------------------
>
> (Updated Oct. 24, 2013, 10:18 p.m.)
>
>
> Review request for Flume.
>
>
> Bugs: FLUME-1227
>     https://issues.apache.org/jira/browse/FLUME-1227
>
>
> Repository: flume-git
>
>
> Description
> -------
>
> Revised design for Spillable Mem Channel.
> We no longer have Spillable channel config pointing to another channel (by name) as in the previous design.
>
> Spillable Channel instead derives from FileChannel (as per https://issues.apache.org/jira/browse/FLUME-1227?focusedCommentId=13628201&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13628201)
>
> Essence of this design:
> - SC derives from File channel and maintains an in memory queue. If memory queue is full, events are sent to disk overflow (i.e. File channel).
> - SC maintains a 'Drain-Order' queue (DOQ) for remembering the order in which the incoming events were interleaved between main memory and
> - Put transaction: All the elements in the putList are committed to mem queue if it has space, else written to disk (ie file channel). Head of DOQ is updated to indicate where the elements put.
> - Take transaction : Tail of DOQ is consulted to determine whether the next set of events are to be taken from the memory queue or from disk overflow. DOQ's tail is updated after events are taken out.
>
>
> SC Configuration:
> - Accepts all the File Channel settings
> - Introduces one additional setting: 'memoryCapacity' which indicates the number of items it can hold in memory
>  
>
> Sample config ...
>
> a1.channels = c1
> a1.sinks = logger
> a1.sources = src
>
> a1.sources.src.type = exec
> a1.sources.src.command = seq 1 100000
> a1.sources.src.batchSize = 10
> a1.sources.src.channels = c1
>
> a1.sinks.logger.type = logger
> a1.sinks.logger.channel = c1
>
> a1.channels.c1.type = spillablememory
> a1.channels.c1.checkpointDir = /tmp/flume/checkpoint
> a1.channels.c1.dataDirs = /tmp/flume/data
> a1.channels.c1.memoryCapacity = 10
> a1.channels.c1.keep-alive = 2
>
>
> Diffs
> -----
>
>   flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java 36f150b
>   flume-ng-channels/flume-spillable-memory-channel/pom.xml PRE-CREATION
>   flume-ng-channels/flume-spillable-memory-channel/src/main/java/org/apache/flume/channel/SpillableMemoryChannel.java PRE-CREATION
>   flume-ng-channels/flume-spillable-memory-channel/src/test/java/org/apache/flume/channel/TestSpillableMemoryChannel.java PRE-CREATION