Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume, mail # user - Short guide to custom configuration provider?


+
Bertrand Dechoux 2013-03-29, 13:53
+
Brock Noland 2013-03-29, 15:19
Copy link to this message
-
Re: Short guide to custom configuration provider?
Bertrand Dechoux 2013-03-29, 15:46
Thanks for the hints!

The part that you quoted is in fact from the flume wiki and I forgot to
quote it myself. But the page is quite old so I wasn't sure about the state
of the feature. I thought maybe commons-configuration was used but wasn't
be able to confirm it clearly (or more exactly I lost myself in this
dependency, which might be used by one of my current projects somehow).

https://cwiki.apache.org/confluence/display/FLUME/Flume+NG

Configuration Provider
> Flume NG has a pluggable configuration system called the configuration
> provider. By default, Flume NG ships with a Java property file based
> configuration system that is both simple and easy to generate
> programmatically. Flume OG has a centralized configuration system with
> a master and ZooKeeper for coordination and we recognize this is very
> appealing to some users where as others see it as overhead they simply
> don't want. We opted to make this a pluggable extension point and ship
> a basic implementation that would let many users get started quickly
> and easily. There's almost certainly enough desire for a similar
> implementation to that of Flume OG, but it isn't yet implemented.
> Users may also implement arbitrary plugins to integrate with any type
> of configuration system (JSON files, a shared RDBMS, a central system
> with ZooKeeper, and so forth). We see this as something more
> interesting to system integrators.
>

I will look at the classes you pointed for my own education and I will see
what I can do.

Thanks a lot.

Bertrand

On Fri, Mar 29, 2013 at 4:19 PM, Brock Noland <[EMAIL PROTECTED]> wrote:

> Hi,
>
> "Flume NG has a pluggable configuration system called the configuration
> provider."
>
> There is an object hierarchy, but I don't consider it pluggable as there
> is no place to "plugin" your own implementation. I consider these things
> "private" to Flume. Meaning that there is no guaranteed API stability
> between releases. With that caveat, you'd want to implement the following
> abstract class:
>
>
> https://github.com/apache/flume/blob/trunk/flume-ng-node/src/main/java/org/apache/flume/node/AbstractConfigurationProvider.java
>
> e.g
> https://github.com/apache/flume/blob/trunk/flume-ng-node/src/main/java/org/apache/flume/node/PropertiesFileConfigurationProvider.java
>
> and then'd you have to customize the Application class to "plugin" your
> own implementation:
>
>
> https://github.com/apache/flume/blob/trunk/flume-ng-node/src/main/java/org/apache/flume/node/Application.java
>
> Brock
>
>
> On Fri, Mar 29, 2013 at 8:53 AM, Bertrand Dechoux <[EMAIL PROTECTED]>wrote:
>
>> Hi,
>>
>> Flume NG has a pluggable configuration system called the configuration
>> provider. I cloned the source repository and I indeed found it but I am a
>> bit lost. Is there some kind of short guide explaining how to write a
>> custom configuration provider?
>>
>> I would like to start with something simple : using a YAML configuration
>> file instead of a properties file.
>> It should be almost trivial to change the following configuration
>>
>> *agent1.channels.ch1.type = memory
>>
>> agent1.sources.avro-source1.channels = ch1
>> agent1.sources.avro-source1.type = avro
>> agent1.sources.avro-source1.bind = 0.0.0.0
>> agent1.sources.avro-source1.port = 41414
>>
>> agent1.sinks.log-sink1.channel = ch1
>> agent1.sinks.log-sink1.type = logger
>>
>> agent1.channels = ch1
>> agent1.sources = avro-source1
>> agent1.sinks = log-sink1*
>>
>> into
>>
>> agent1:
>>
>>     channels:
>>         ch1:
>>             type: memory
>>
>>     sources:
>>         avroSource1:
>>             channels: ch1
>>             type: avro
>>             bind: 0.0.0.0
>>             port: 41414
>>
>>     sinks:
>>         logSink1:
>>             channel: ch1
>>             type: logger
>>
>>
>> At least, I know how to convert the latter into the former but I am not
>> sure how this transformation can be included into flume-ng using a
>> configuration provider.
>> I have not been able to find much information about the customisation of
+
Bertrand Dechoux 2013-03-29, 16:43
+
Hari Shreedharan 2013-03-29, 19:06
+
Bertrand Dechoux 2013-04-02, 19:23
+
Hari Shreedharan 2013-04-02, 20:34