Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # user >> Short guide to custom configuration provider?

Copy link to this message
Re: Short guide to custom configuration provider?
Thanks for the hints!

The part that you quoted is in fact from the flume wiki and I forgot to
quote it myself. But the page is quite old so I wasn't sure about the state
of the feature. I thought maybe commons-configuration was used but wasn't
be able to confirm it clearly (or more exactly I lost myself in this
dependency, which might be used by one of my current projects somehow).


Configuration Provider
> Flume NG has a pluggable configuration system called the configuration
> provider. By default, Flume NG ships with a Java property file based
> configuration system that is both simple and easy to generate
> programmatically. Flume OG has a centralized configuration system with
> a master and ZooKeeper for coordination and we recognize this is very
> appealing to some users where as others see it as overhead they simply
> don't want. We opted to make this a pluggable extension point and ship
> a basic implementation that would let many users get started quickly
> and easily. There's almost certainly enough desire for a similar
> implementation to that of Flume OG, but it isn't yet implemented.
> Users may also implement arbitrary plugins to integrate with any type
> of configuration system (JSON files, a shared RDBMS, a central system
> with ZooKeeper, and so forth). We see this as something more
> interesting to system integrators.

I will look at the classes you pointed for my own education and I will see
what I can do.

Thanks a lot.


On Fri, Mar 29, 2013 at 4:19 PM, Brock Noland <[EMAIL PROTECTED]> wrote:

> Hi,
> "Flume NG has a pluggable configuration system called the configuration
> provider."
> There is an object hierarchy, but I don't consider it pluggable as there
> is no place to "plugin" your own implementation. I consider these things
> "private" to Flume. Meaning that there is no guaranteed API stability
> between releases. With that caveat, you'd want to implement the following
> abstract class:
> https://github.com/apache/flume/blob/trunk/flume-ng-node/src/main/java/org/apache/flume/node/AbstractConfigurationProvider.java
> e.g
> https://github.com/apache/flume/blob/trunk/flume-ng-node/src/main/java/org/apache/flume/node/PropertiesFileConfigurationProvider.java
> and then'd you have to customize the Application class to "plugin" your
> own implementation:
> https://github.com/apache/flume/blob/trunk/flume-ng-node/src/main/java/org/apache/flume/node/Application.java
> Brock
> On Fri, Mar 29, 2013 at 8:53 AM, Bertrand Dechoux <[EMAIL PROTECTED]>wrote:
>> Hi,
>> Flume NG has a pluggable configuration system called the configuration
>> provider. I cloned the source repository and I indeed found it but I am a
>> bit lost. Is there some kind of short guide explaining how to write a
>> custom configuration provider?
>> I would like to start with something simple : using a YAML configuration
>> file instead of a properties file.
>> It should be almost trivial to change the following configuration
>> *agent1.channels.ch1.type = memory
>> agent1.sources.avro-source1.channels = ch1
>> agent1.sources.avro-source1.type = avro
>> agent1.sources.avro-source1.bind =
>> agent1.sources.avro-source1.port = 41414
>> agent1.sinks.log-sink1.channel = ch1
>> agent1.sinks.log-sink1.type = logger
>> agent1.channels = ch1
>> agent1.sources = avro-source1
>> agent1.sinks = log-sink1*
>> into
>> agent1:
>>     channels:
>>         ch1:
>>             type: memory
>>     sources:
>>         avroSource1:
>>             channels: ch1
>>             type: avro
>>             bind:
>>             port: 41414
>>     sinks:
>>         logSink1:
>>             channel: ch1
>>             type: logger
>> At least, I know how to convert the latter into the former but I am not
>> sure how this transformation can be included into flume-ng using a
>> configuration provider.
>> I have not been able to find much information about the customisation of