Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume, mail # user - Short guide to custom configuration provider?


Copy link to this message
-
Re: Short guide to custom configuration provider?
Hari Shreedharan 2013-03-29, 19:06
I actually do like the idea of pluggable config provider. Like you mentioned, it should be possible to hack the Application class, and use a CLI arg to select the config provider. If no CLI arg is provided, just fall back to the current mechanism. If you submit a patch, I'd be glad to review it :-)
Thanks
Hari

--
Hari Shreedharan
On Friday, March 29, 2013 at 9:43 AM, Bertrand Dechoux wrote:

> It seems quite easier than dealing with commons configuration.
>
> I might take a shot at it during the week-end, if I find the time.
>
> About the customisation of the Application class, that's something which is already mostly done by
> https://issues.apache.org/jira/browse/FLUME-1491 : Dynamic Configuration from Zookeeper watcher
>
> It might be a good idea to split this improvement into :
> 1) Selection of an alternative configuration provider
> 2) A zookeeper based configuration provider
>
> If point 1 is done, anyone could use her/his custom configuration provider by dropping the implementation as a third party jar and then selecting it with the help of the command line.
>
> Bertrand
>
>
> On Fri, Mar 29, 2013 at 4:46 PM, Bertrand Dechoux <[EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])> wrote:
> > Thanks for the hints!
> >
> > The part that you quoted is in fact from the flume wiki and I forgot to quote it myself. But the page is quite old so I wasn't sure about the state of the feature. I thought maybe commons-configuration was used but wasn't be able to confirm it clearly (or more exactly I lost myself in this dependency, which might be used by one of my current projects somehow).
> >
> > https://cwiki.apache.org/confluence/display/FLUME/Flume+NG
> >
> > > Configuration Provider
> > >
> > > Flume NG has a pluggable configuration system called the configuration
> > > provider. By default, Flume NG ships with a Java property file based
> > > configuration system that is both simple and easy to generate
> > > programmatically. Flume OG has a centralized configuration system with
> > > a master and ZooKeeper for coordination and we recognize this is very
> > > appealing to some users where as others see it as overhead they simply
> > > don't want. We opted to make this a pluggable extension point and ship
> > > a basic implementation that would let many users get started quickly
> > > and easily. There's almost certainly enough desire for a similar
> > > implementation to that of Flume OG, but it isn't yet implemented.
> > > Users may also implement arbitrary plugins to integrate with any type
> > > of configuration system (JSON files, a shared RDBMS, a central system
> > > with ZooKeeper, and so forth). We see this as something more
> > > interesting to system integrators.
> >
> > I will look at the classes you pointed for my own education and I will see what I can do.
> >
> > Thanks a lot.
> >
> > Bertrand
> >
> >
> > On Fri, Mar 29, 2013 at 4:19 PM, Brock Noland <[EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])> wrote:
> > > Hi,
> > >
> > > "Flume NG has a pluggable configuration system called the configuration provider."
> > >
> > > There is an object hierarchy, but I don't consider it pluggable as there is no place to "plugin" your own implementation. I consider these things "private" to Flume. Meaning that there is no guaranteed API stability between releases. With that caveat, you'd want to implement the following abstract class:
> > >
> > > https://github.com/apache/flume/blob/trunk/flume-ng-node/src/main/java/org/apache/flume/node/AbstractConfigurationProvider.java
> > >
> > > e.g https://github.com/apache/flume/blob/trunk/flume-ng-node/src/main/java/org/apache/flume/node/PropertiesFileConfigurationProvider.java
> > >
> > > and then'd you have to customize the Application class to "plugin" your own implementation:
> > >
> > > https://github.com/apache/flume/blob/trunk/flume-ng-node/src/main/java/org/apache/flume/node/Application.java
> > >
> > > Brock
> > >
> > >
> > > On Fri, Mar 29, 2013 at 8:53 AM, Bertrand Dechoux <[EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])> wrote: