Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # user >> Short guide to custom configuration provider?


Copy link to this message
-
Re: Short guide to custom configuration provider?
Thanks for the hints!

The part that you quoted is in fact from the flume wiki and I forgot to
quote it myself. But the page is quite old so I wasn't sure about the state
of the feature. I thought maybe commons-configuration was used but wasn't
be able to confirm it clearly (or more exactly I lost myself in this
dependency, which might be used by one of my current projects somehow).

https://cwiki.apache.org/confluence/display/FLUME/Flume+NG

Configuration Provider
> Flume NG has a pluggable configuration system called the configuration
> provider. By default, Flume NG ships with a Java property file based
> configuration system that is both simple and easy to generate
> programmatically. Flume OG has a centralized configuration system with
> a master and ZooKeeper for coordination and we recognize this is very
> appealing to some users where as others see it as overhead they simply
> don't want. We opted to make this a pluggable extension point and ship
> a basic implementation that would let many users get started quickly
> and easily. There's almost certainly enough desire for a similar
> implementation to that of Flume OG, but it isn't yet implemented.
> Users may also implement arbitrary plugins to integrate with any type
> of configuration system (JSON files, a shared RDBMS, a central system
> with ZooKeeper, and so forth). We see this as something more
> interesting to system integrators.
>

I will look at the classes you pointed for my own education and I will see
what I can do.

Thanks a lot.

Bertrand

On Fri, Mar 29, 2013 at 4:19 PM, Brock Noland <[EMAIL PROTECTED]> wrote:

> Hi,
>
> "Flume NG has a pluggable configuration system called the configuration
> provider."
>
> There is an object hierarchy, but I don't consider it pluggable as there
> is no place to "plugin" your own implementation. I consider these things
> "private" to Flume. Meaning that there is no guaranteed API stability
> between releases. With that caveat, you'd want to implement the following
> abstract class:
>
>
> https://github.com/apache/flume/blob/trunk/flume-ng-node/src/main/java/org/apache/flume/node/AbstractConfigurationProvider.java
>
> e.g
> https://github.com/apache/flume/blob/trunk/flume-ng-node/src/main/java/org/apache/flume/node/PropertiesFileConfigurationProvider.java
>
> and then'd you have to customize the Application class to "plugin" your
> own implementation:
>
>
> https://github.com/apache/flume/blob/trunk/flume-ng-node/src/main/java/org/apache/flume/node/Application.java
>
> Brock
>
>
> On Fri, Mar 29, 2013 at 8:53 AM, Bertrand Dechoux <[EMAIL PROTECTED]>wrote:
>
>> Hi,
>>
>> Flume NG has a pluggable configuration system called the configuration
>> provider. I cloned the source repository and I indeed found it but I am a
>> bit lost. Is there some kind of short guide explaining how to write a
>> custom configuration provider?
>>
>> I would like to start with something simple : using a YAML configuration
>> file instead of a properties file.
>> It should be almost trivial to change the following configuration
>>
>> *agent1.channels.ch1.type = memory
>>
>> agent1.sources.avro-source1.channels = ch1
>> agent1.sources.avro-source1.type = avro
>> agent1.sources.avro-source1.bind = 0.0.0.0
>> agent1.sources.avro-source1.port = 41414
>>
>> agent1.sinks.log-sink1.channel = ch1
>> agent1.sinks.log-sink1.type = logger
>>
>> agent1.channels = ch1
>> agent1.sources = avro-source1
>> agent1.sinks = log-sink1*
>>
>> into
>>
>> agent1:
>>
>>     channels:
>>         ch1:
>>             type: memory
>>
>>     sources:
>>         avroSource1:
>>             channels: ch1
>>             type: avro
>>             bind: 0.0.0.0
>>             port: 41414
>>
>>     sinks:
>>         logSink1:
>>             channel: ch1
>>             type: logger
>>
>>
>> At least, I know how to convert the latter into the former but I am not
>> sure how this transformation can be included into flume-ng using a
>> configuration provider.
>> I have not been able to find much information about the customisation of
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB