Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # user >> Short guide to custom configuration provider?


Copy link to this message
-
Re: Short guide to custom configuration provider?
It seems quite easier than dealing with commons configuration.

I might take a shot at it during the week-end, if I find the time.

About the customisation of the Application class, that's something which is
already mostly done by
https://issues.apache.org/jira/browse/FLUME-1491 : Dynamic Configuration
from Zookeeper watcher

It might be a good idea to split this improvement into :
1) Selection of an alternative configuration provider
2) A zookeeper based configuration provider

If point 1 is done, anyone could use her/his custom configuration provider
by dropping the implementation as a third party jar and then selecting it
with the help of the command line.

Bertrand
On Fri, Mar 29, 2013 at 4:46 PM, Bertrand Dechoux <[EMAIL PROTECTED]>wrote:

> Thanks for the hints!
>
> The part that you quoted is in fact from the flume wiki and I forgot to
> quote it myself. But the page is quite old so I wasn't sure about the state
> of the feature. I thought maybe commons-configuration was used but wasn't
> be able to confirm it clearly (or more exactly I lost myself in this
> dependency, which might be used by one of my current projects somehow).
>
> https://cwiki.apache.org/confluence/display/FLUME/Flume+NG
>
> Configuration Provider
>>
>> Flume NG has a pluggable configuration system called the configuration
>> provider. By default, Flume NG ships with a Java property file based
>> configuration system that is both simple and easy to generate
>> programmatically. Flume OG has a centralized configuration system with
>> a master and ZooKeeper for coordination and we recognize this is very
>> appealing to some users where as others see it as overhead they simply
>> don't want. We opted to make this a pluggable extension point and ship
>> a basic implementation that would let many users get started quickly
>> and easily. There's almost certainly enough desire for a similar
>> implementation to that of Flume OG, but it isn't yet implemented.
>> Users may also implement arbitrary plugins to integrate with any type
>> of configuration system (JSON files, a shared RDBMS, a central system
>> with ZooKeeper, and so forth). We see this as something more
>> interesting to system integrators.
>>
>
> I will look at the classes you pointed for my own education and I will see
> what I can do.
>
> Thanks a lot.
>
> Bertrand
>
>
> On Fri, Mar 29, 2013 at 4:19 PM, Brock Noland <[EMAIL PROTECTED]> wrote:
>
>> Hi,
>>
>> "Flume NG has a pluggable configuration system called the configuration
>> provider."
>>
>> There is an object hierarchy, but I don't consider it pluggable as there
>> is no place to "plugin" your own implementation. I consider these things
>> "private" to Flume. Meaning that there is no guaranteed API stability
>> between releases. With that caveat, you'd want to implement the following
>> abstract class:
>>
>>
>> https://github.com/apache/flume/blob/trunk/flume-ng-node/src/main/java/org/apache/flume/node/AbstractConfigurationProvider.java
>>
>> e.g
>> https://github.com/apache/flume/blob/trunk/flume-ng-node/src/main/java/org/apache/flume/node/PropertiesFileConfigurationProvider.java
>>
>> and then'd you have to customize the Application class to "plugin" your
>> own implementation:
>>
>>
>> https://github.com/apache/flume/blob/trunk/flume-ng-node/src/main/java/org/apache/flume/node/Application.java
>>
>> Brock
>>
>>
>> On Fri, Mar 29, 2013 at 8:53 AM, Bertrand Dechoux <[EMAIL PROTECTED]>wrote:
>>
>>> Hi,
>>>
>>> Flume NG has a pluggable configuration system called the configuration
>>> provider. I cloned the source repository and I indeed found it but I am a
>>> bit lost. Is there some kind of short guide explaining how to write a
>>> custom configuration provider?
>>>
>>> I would like to start with something simple : using a YAML configuration
>>> file instead of a properties file.
>>> It should be almost trivial to change the following configuration
>>>
>>> *agent1.channels.ch1.type = memory
>>>
>>> agent1.sources.avro-source1.channels = ch1
Bertrand Dechoux