Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # dev >> Review Request: Tools to generate configuration file


Copy link to this message
-
Re: Review Request: Tools to generate configuration file

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/6206/
-----------------------------------------------------------

(Updated July 31, 2012, 1:08 a.m.)
Review request for Flume and Brock Noland.
Changes
-------

Updated two classes structures to avoid invalid source code description
Description
-------

Here is the tool for generating Flume configuration file with Java code.
This addresses bug https://issues.apache.org/jira/browse/FLUME-1405.
    https://issues.apache.org/jira/browse/https://issues.apache.org/jira/browse/FLUME-1405
Diffs (updated)
-----

  flume/trunk/flume-ng-configuration/src/main/java/org/apache/flume/conf/gen/AbstractComponent.java PRE-CREATION
  flume/trunk/flume-ng-configuration/src/main/java/org/apache/flume/conf/gen/Agent.java PRE-CREATION
  flume/trunk/flume-ng-configuration/src/main/java/org/apache/flume/conf/gen/Channel.java PRE-CREATION
  flume/trunk/flume-ng-configuration/src/main/java/org/apache/flume/conf/gen/ConfGenerator.java PRE-CREATION
  flume/trunk/flume-ng-configuration/src/main/java/org/apache/flume/conf/gen/Interceptor.java PRE-CREATION
  flume/trunk/flume-ng-configuration/src/main/java/org/apache/flume/conf/gen/NameAvailable.java PRE-CREATION
  flume/trunk/flume-ng-configuration/src/main/java/org/apache/flume/conf/gen/Recorder.java PRE-CREATION
  flume/trunk/flume-ng-configuration/src/main/java/org/apache/flume/conf/gen/Sink.java PRE-CREATION
  flume/trunk/flume-ng-configuration/src/main/java/org/apache/flume/conf/gen/SinkGroup.java PRE-CREATION
  flume/trunk/flume-ng-configuration/src/main/java/org/apache/flume/conf/gen/Source.java PRE-CREATION
  flume/trunk/flume-ng-configuration/src/main/java/org/apache/flume/conf/gen/Validation.java PRE-CREATION

Diff: https://reviews.apache.org/r/6206/diff/
Testing
-------

Test example:

Agent agent = new Agent().setName("agent");

Source source = new Source("scribe")
                .setProperty("port", "1499")
                .setType(SourceType.OTHER,"org.apache.flume.source.scribe.ScribeSource");

agent.registerSource(source);

for (int i = 0; i < 5; i++) {
  Channel channel = new Channel("mc" + i)
                .setProperty("capacity", "1000000")
                .setType(ChannelType.MEMORY, null);

  agent.registerChannel(channel);

  Sink sink = new Sink("hfds" + i)
                .setType(SinkType.HDFS, null)
                .setProperty("hfds.txtEventMax", "1000")
                .setProperty("hdfs.path", "/flume_test/data" + i)
                .connectChannel(channel);

  agent.registerSink(sink);
  source.connectChannel(channel);
}

ConfGenerator.generateConf(agent, new File("XXXXX"));
Thanks,

Denny Ye