Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume, mail # dev - Proposal for tools to generate duplicated configuration


Copy link to this message
-
Proposal for tools to generate duplicated configuration
Denny Ye 2012-07-26, 06:12
hi all,
    In the progress of maintaining Flume NG, it's hard to manage
configuration in too many flows.
    Meanwhile, most of such configurations are similar. Maybe tiny
difference in component name;

    Thus, I would like to generate configuration with Java code in simple
method.
    I created and tested tool to generate configuration. That runs
regularly.

    Could anyone give your tips or advice? If tool like this is useful for
you, I will submit it to community.
    Wish your attention, thanks!
     Java Code Example:
                Agent agent = new Agent().setName("agent");
 Source source = new Source("scribe")
.setProperty("port", "1499")
.setType(SourceType.OTHER, "org.apache.flume.source.scribe.ScribeSource");
agent.registerSource(source);
 for (int i = 0; i < 5; i++) {
Channel channel = new Channel("mc" + i)
      .setProperty("capacity", "1000000")
      .setType(ChannelType.MEMORY, null);
agent.registerChannel(channel);
 Sink sink = new Sink("hfds" + i)
      .setType(SinkType.HDFS, null)
      .setProperty("hfds.txtEventMax", "1000")
       .setProperty("hdfs.path", "/flume_test/data" + i)
       .connectChannel(channel);
agent.registerSink(sink);
 source.connectChannel(channel);
}

                ConfGenerator.generateConf(agent, new
File("d:\\flume.conf"));

-Regards
Denny Ye