Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume >> mail # user >> Short guide to custom configuration provider?


+
Bertrand Dechoux 2013-03-29, 13:53
+
Brock Noland 2013-03-29, 15:19
+
Bertrand Dechoux 2013-03-29, 15:46
+
Bertrand Dechoux 2013-03-29, 16:43
+
Hari Shreedharan 2013-03-29, 19:06
Copy link to this message
-
Re: Short guide to custom configuration provider?
I did a short proof of concept for the YAML part here
https://github.com/BertrandDechoux/flume-ng-yaml

Does your reaction mean that FLUME-1491 is a 'zombie'? Anyway, I will look
at what was done in that issue and see what can be done for a clean
selection of a custom configuration provider.

Bertrand
On Fri, Mar 29, 2013 at 8:06 PM, Hari Shreedharan <[EMAIL PROTECTED]
> wrote:

>  I actually do like the idea of pluggable config provider. Like you
> mentioned, it should be possible to hack the Application class, and use a
> CLI arg to select the config provider. If no CLI arg is provided, just fall
> back to the current mechanism. If you submit a patch, I'd be glad to review
> it :-)
>
>
> Thanks
> Hari
>
> --
> Hari Shreedharan
>
> On Friday, March 29, 2013 at 9:43 AM, Bertrand Dechoux wrote:
>
> It seems quite easier than dealing with commons configuration.
>
> I might take a shot at it during the week-end, if I find the time.
>
> About the customisation of the Application class, that's something which
> is already mostly done by
> https://issues.apache.org/jira/browse/FLUME-1491 : Dynamic Configuration
> from Zookeeper watcher
>
> It might be a good idea to split this improvement into :
> 1) Selection of an alternative configuration provider
> 2) A zookeeper based configuration provider
>
> If point 1 is done, anyone could use her/his custom configuration provider
> by dropping the implementation as a third party jar and then selecting it
> with the help of the command line.
>
> Bertrand
>
>
> On Fri, Mar 29, 2013 at 4:46 PM, Bertrand Dechoux <[EMAIL PROTECTED]>wrote:
>
> Thanks for the hints!
>
> The part that you quoted is in fact from the flume wiki and I forgot to
> quote it myself. But the page is quite old so I wasn't sure about the state
> of the feature. I thought maybe commons-configuration was used but wasn't
> be able to confirm it clearly (or more exactly I lost myself in this
> dependency, which might be used by one of my current projects somehow).
>
> https://cwiki.apache.org/confluence/display/FLUME/Flume+NG
>
> Configuration Provider
>
> Flume NG has a pluggable configuration system called the configuration
> provider. By default, Flume NG ships with a Java property file based
> configuration system that is both simple and easy to generate
> programmatically. Flume OG has a centralized configuration system with
> a master and ZooKeeper for coordination and we recognize this is very
> appealing to some users where as others see it as overhead they simply
> don't want. We opted to make this a pluggable extension point and ship
> a basic implementation that would let many users get started quickly
> and easily. There's almost certainly enough desire for a similar
> implementation to that of Flume OG, but it isn't yet implemented.
> Users may also implement arbitrary plugins to integrate with any type
> of configuration system (JSON files, a shared RDBMS, a central system
> with ZooKeeper, and so forth). We see this as something more
> interesting to system integrators.
>
>
> I will look at the classes you pointed for my own education and I will see
> what I can do.
>
> Thanks a lot.
>
> Bertrand
>
>
> On Fri, Mar 29, 2013 at 4:19 PM, Brock Noland <[EMAIL PROTECTED]> wrote:
>
> Hi,
>
> "Flume NG has a pluggable configuration system called the configuration
> provider."
>
> There is an object hierarchy, but I don't consider it pluggable as there
> is no place to "plugin" your own implementation. I consider these things
> "private" to Flume. Meaning that there is no guaranteed API stability
> between releases. With that caveat, you'd want to implement the following
> abstract class:
>
>
> https://github.com/apache/flume/blob/trunk/flume-ng-node/src/main/java/org/apache/flume/node/AbstractConfigurationProvider.java
>
> e.g
> https://github.com/apache/flume/blob/trunk/flume-ng-node/src/main/java/org/apache/flume/node/PropertiesFileConfigurationProvider.java
>
> and then'd you have to customize the Application class to "plugin" your
Bertrand Dechoux
+
Hari Shreedharan 2013-04-02, 20:34
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB