Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # user >> Short guide to custom configuration provider?

Copy link to this message
Re: Short guide to custom configuration provider?
It seems quite easier than dealing with commons configuration.

I might take a shot at it during the week-end, if I find the time.

About the customisation of the Application class, that's something which is
already mostly done by
https://issues.apache.org/jira/browse/FLUME-1491 : Dynamic Configuration
from Zookeeper watcher

It might be a good idea to split this improvement into :
1) Selection of an alternative configuration provider
2) A zookeeper based configuration provider

If point 1 is done, anyone could use her/his custom configuration provider
by dropping the implementation as a third party jar and then selecting it
with the help of the command line.

On Fri, Mar 29, 2013 at 4:46 PM, Bertrand Dechoux <[EMAIL PROTECTED]>wrote:

> Thanks for the hints!
> The part that you quoted is in fact from the flume wiki and I forgot to
> quote it myself. But the page is quite old so I wasn't sure about the state
> of the feature. I thought maybe commons-configuration was used but wasn't
> be able to confirm it clearly (or more exactly I lost myself in this
> dependency, which might be used by one of my current projects somehow).
> https://cwiki.apache.org/confluence/display/FLUME/Flume+NG
> Configuration Provider
>> Flume NG has a pluggable configuration system called the configuration
>> provider. By default, Flume NG ships with a Java property file based
>> configuration system that is both simple and easy to generate
>> programmatically. Flume OG has a centralized configuration system with
>> a master and ZooKeeper for coordination and we recognize this is very
>> appealing to some users where as others see it as overhead they simply
>> don't want. We opted to make this a pluggable extension point and ship
>> a basic implementation that would let many users get started quickly
>> and easily. There's almost certainly enough desire for a similar
>> implementation to that of Flume OG, but it isn't yet implemented.
>> Users may also implement arbitrary plugins to integrate with any type
>> of configuration system (JSON files, a shared RDBMS, a central system
>> with ZooKeeper, and so forth). We see this as something more
>> interesting to system integrators.
> I will look at the classes you pointed for my own education and I will see
> what I can do.
> Thanks a lot.
> Bertrand
> On Fri, Mar 29, 2013 at 4:19 PM, Brock Noland <[EMAIL PROTECTED]> wrote:
>> Hi,
>> "Flume NG has a pluggable configuration system called the configuration
>> provider."
>> There is an object hierarchy, but I don't consider it pluggable as there
>> is no place to "plugin" your own implementation. I consider these things
>> "private" to Flume. Meaning that there is no guaranteed API stability
>> between releases. With that caveat, you'd want to implement the following
>> abstract class:
>> https://github.com/apache/flume/blob/trunk/flume-ng-node/src/main/java/org/apache/flume/node/AbstractConfigurationProvider.java
>> e.g
>> https://github.com/apache/flume/blob/trunk/flume-ng-node/src/main/java/org/apache/flume/node/PropertiesFileConfigurationProvider.java
>> and then'd you have to customize the Application class to "plugin" your
>> own implementation:
>> https://github.com/apache/flume/blob/trunk/flume-ng-node/src/main/java/org/apache/flume/node/Application.java
>> Brock
>> On Fri, Mar 29, 2013 at 8:53 AM, Bertrand Dechoux <[EMAIL PROTECTED]>wrote:
>>> Hi,
>>> Flume NG has a pluggable configuration system called the configuration
>>> provider. I cloned the source repository and I indeed found it but I am a
>>> bit lost. Is there some kind of short guide explaining how to write a
>>> custom configuration provider?
>>> I would like to start with something simple : using a YAML configuration
>>> file instead of a properties file.
>>> It should be almost trivial to change the following configuration
>>> *agent1.channels.ch1.type = memory
>>> agent1.sources.avro-source1.channels = ch1
Bertrand Dechoux