Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # dev >> Proposal for tools to generate duplicated configuration


Copy link to this message
-
Proposal for tools to generate duplicated configuration
hi all,
    In the progress of maintaining Flume NG, it's hard to manage
configuration in too many flows.
    Meanwhile, most of such configurations are similar. Maybe tiny
difference in component name;

    Thus, I would like to generate configuration with Java code in simple
method.
    I created and tested tool to generate configuration. That runs
regularly.

    Could anyone give your tips or advice? If tool like this is useful for
you, I will submit it to community.
    Wish your attention, thanks!
     Java Code Example:
                Agent agent = new Agent().setName("agent");
 Source source = new Source("scribe")
.setProperty("port", "1499")
.setType(SourceType.OTHER, "org.apache.flume.source.scribe.ScribeSource");
agent.registerSource(source);
 for (int i = 0; i < 5; i++) {
Channel channel = new Channel("mc" + i)
      .setProperty("capacity", "1000000")
      .setType(ChannelType.MEMORY, null);
agent.registerChannel(channel);
 Sink sink = new Sink("hfds" + i)
      .setType(SinkType.HDFS, null)
      .setProperty("hfds.txtEventMax", "1000")
       .setProperty("hdfs.path", "/flume_test/data" + i)
       .connectChannel(channel);
agent.registerSink(sink);
 source.connectChannel(channel);
}

                ConfGenerator.generateConf(agent, new
File("d:\\flume.conf"));

-Regards
Denny Ye