Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume, mail # user - Short guide to custom configuration provider?


Copy link to this message
-
Re: Short guide to custom configuration provider?
Brock Noland 2013-03-29, 15:19
Hi,

"Flume NG has a pluggable configuration system called the configuration
provider."

There is an object hierarchy, but I don't consider it pluggable as there is
no place to "plugin" your own implementation. I consider these things
"private" to Flume. Meaning that there is no guaranteed API stability
between releases. With that caveat, you'd want to implement the following
abstract class:

https://github.com/apache/flume/blob/trunk/flume-ng-node/src/main/java/org/apache/flume/node/AbstractConfigurationProvider.java

e.g
https://github.com/apache/flume/blob/trunk/flume-ng-node/src/main/java/org/apache/flume/node/PropertiesFileConfigurationProvider.java

and then'd you have to customize the Application class to "plugin" your own
implementation:

https://github.com/apache/flume/blob/trunk/flume-ng-node/src/main/java/org/apache/flume/node/Application.java

Brock
On Fri, Mar 29, 2013 at 8:53 AM, Bertrand Dechoux <[EMAIL PROTECTED]>wrote:

> Hi,
>
> Flume NG has a pluggable configuration system called the configuration
> provider. I cloned the source repository and I indeed found it but I am a
> bit lost. Is there some kind of short guide explaining how to write a
> custom configuration provider?
>
> I would like to start with something simple : using a YAML configuration
> file instead of a properties file.
> It should be almost trivial to change the following configuration
>
> *agent1.channels.ch1.type = memory
>
> agent1.sources.avro-source1.channels = ch1
> agent1.sources.avro-source1.type = avro
> agent1.sources.avro-source1.bind = 0.0.0.0
> agent1.sources.avro-source1.port = 41414
>
> agent1.sinks.log-sink1.channel = ch1
> agent1.sinks.log-sink1.type = logger
>
> agent1.channels = ch1
> agent1.sources = avro-source1
> agent1.sinks = log-sink1*
>
> into
>
> agent1:
>
>     channels:
>         ch1:
>             type: memory
>
>     sources:
>         avroSource1:
>             channels: ch1
>             type: avro
>             bind: 0.0.0.0
>             port: 41414
>
>     sinks:
>         logSink1:
>             channel: ch1
>             type: logger
>
>
> At least, I know how to convert the latter into the former but I am not
> sure how this transformation can be included into flume-ng using a
> configuration provider.
> I have not been able to find much information about the customisation of
> the configuration provider. Did I miss any documentation?
>
>
> Bertrand
>

--
Apache MRUnit - Unit testing MapReduce - http://mrunit.apache.org