Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume, mail # dev - Re: Review Request 12060: Revised design for Spillable Mem Channel

Roshan Naik 2013-07-03, 23:56
Hari Shreedharan 2013-08-30, 20:41
Roshan Naik 2013-10-04, 21:49
Roshan Naik 2013-10-15, 20:44
Hari Shreedharan 2013-10-16, 00:06
Roshan Naik 2013-10-24, 22:16
Copy link to this message
Re: Review Request 12060: Revised design for Spillable Mem Channel
Roshan Naik 2013-10-24, 22:16

This is an automatically generated e-mail. To reply, visit:

(Updated Oct. 24, 2013, 10:16 p.m.)
Review request for Flume.

Addressing review comments.
Bugs: FLUME-1227
Repository: flume-git

Revised design for Spillable Mem Channel.
We no longer have Spillable channel config pointing to another channel (by name) as in the previous design.

Spillable Channel instead derives from FileChannel (as per https://issues.apache.org/jira/browse/FLUME-1227?focusedCommentId=13628201&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13628201)

Essence of this design:
- SC derives from File channel and maintains an in memory queue. If memory queue is full, events are sent to disk overflow (i.e. File channel).
- SC maintains a 'Drain-Order' queue (DOQ) for remembering the order in which the incoming events were interleaved between main memory and
- Put transaction: All the elements in the putList are committed to mem queue if it has space, else written to disk (ie file channel). Head of DOQ is updated to indicate where the elements put.
- Take transaction : Tail of DOQ is consulted to determine whether the next set of events are to be taken from the memory queue or from disk overflow. DOQ's tail is updated after events are taken out.
SC Configuration:
- Accepts all the File Channel settings
- Introduces one additional setting: 'memoryCapacity' which indicates the number of items it can hold in memory

Sample config ...

a1.channels = c1
a1.sinks = logger
a1.sources = src

a1.sources.src.type = exec
a1.sources.src.command = seq 1 100000
a1.sources.src.batchSize = 10
a1.sources.src.channels = c1

a1.sinks.logger.type = logger
a1.sinks.logger.channel = c1

a1.channels.c1.type = spillablememory
a1.channels.c1.checkpointDir = /tmp/flume/checkpoint
a1.channels.c1.dataDirs = /tmp/flume/data
a1.channels.c1.memoryCapacity = 10
a1.channels.c1.keep-alive = 2
Diffs (updated)

  flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java 36f150b
  flume-ng-channels/flume-spillable-memory-channel/pom.xml PRE-CREATION
  flume-ng-channels/flume-spillable-memory-channel/src/main/java/org/apache/flume/channel/SpillableMemoryChannel.java PRE-CREATION
  flume-ng-channels/flume-spillable-memory-channel/src/test/java/org/apache/flume/channel/TestSpillableMemoryChannel.java PRE-CREATION
  flume-ng-channels/pom.xml 5832ab4
  flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelConfiguration.java 26f4dd7
  flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelType.java 15b8cc3
  flume-ng-core/src/main/java/org/apache/flume/ChannelFullException.java PRE-CREATION
  flume-ng-core/src/main/java/org/apache/flume/channel/AbstractChannel.java 1370e66
  flume-ng-core/src/main/java/org/apache/flume/channel/MemoryChannel.java 688323d
  flume-ng-dist/pom.xml 2d0ee47
  flume-ng-doc/sphinx/FlumeUserGuide.rst 98859ce
  flume-ng-embedded-agent/src/main/java/org/apache/flume/agent/embedded/EmbeddedAgentConfiguration.java 6204bc5
  flume-ng-node/pom.xml f1b0c65
  pom.xml 267925f

Diff: https://reviews.apache.org/r/12060/diff/

Wrote a set of Unit tests. A few are failing and need to be finished up.
File Attachments

Revised design doc

Roshan Naik

Roshan Naik 2013-10-24, 22:18
Hari Shreedharan 2013-10-31, 01:48
Roshan Naik 2013-10-31, 23:32
Roshan Naik 2013-10-31, 23:59
Roshan Naik 2013-11-05, 03:34
Roshan Naik 2013-12-18, 11:43
Hari Shreedharan 2013-10-31, 19:32