Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume >> mail # user >> Short guide to custom configuration provider?


+
Bertrand Dechoux 2013-03-29, 13:53
+
Brock Noland 2013-03-29, 15:19
+
Bertrand Dechoux 2013-03-29, 15:46
+
Bertrand Dechoux 2013-03-29, 16:43
+
Hari Shreedharan 2013-03-29, 19:06
+
Bertrand Dechoux 2013-04-02, 19:23
Copy link to this message
-
Re: Short guide to custom configuration provider?
Hi Bertrand,

https://issues.apache.org/jira/browse/FLUME-1630 changed a lot of the config system, and made it much easier to write a config provider. Once you write a provider you can update the application class to use reflection to instantiate the provider. I dont remember how the patch on FLUME-1491 looks like, but if it is possible to integrate it into this, it might make sense.
Thanks
Hari

--
Hari Shreedharan
On Tuesday, April 2, 2013 at 12:23 PM, Bertrand Dechoux wrote:

> I did a short proof of concept for the YAML part here
> https://github.com/BertrandDechoux/flume-ng-yaml
>
> Does your reaction mean that FLUME-1491 is a 'zombie'? Anyway, I will look at what was done in that issue and see what can be done for a clean selection of a custom configuration provider.
>
> Bertrand
>
>
> On Fri, Mar 29, 2013 at 8:06 PM, Hari Shreedharan <[EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])> wrote:
> > I actually do like the idea of pluggable config provider. Like you mentioned, it should be possible to hack the Application class, and use a CLI arg to select the config provider. If no CLI arg is provided, just fall back to the current mechanism. If you submit a patch, I'd be glad to review it :-)
> >
> >
> > Thanks
> > Hari
> >
> > --
> > Hari Shreedharan
> >
> >
> > On Friday, March 29, 2013 at 9:43 AM, Bertrand Dechoux wrote:
> >
> > > It seems quite easier than dealing with commons configuration.
> > >
> > > I might take a shot at it during the week-end, if I find the time.
> > >
> > > About the customisation of the Application class, that's something which is already mostly done by
> > > https://issues.apache.org/jira/browse/FLUME-1491 : Dynamic Configuration from Zookeeper watcher
> > >
> > > It might be a good idea to split this improvement into :
> > > 1) Selection of an alternative configuration provider
> > > 2) A zookeeper based configuration provider
> > >
> > > If point 1 is done, anyone could use her/his custom configuration provider by dropping the implementation as a third party jar and then selecting it with the help of the command line.
> > >
> > > Bertrand
> > >
> > >
> > > On Fri, Mar 29, 2013 at 4:46 PM, Bertrand Dechoux <[EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])> wrote:
> > > > Thanks for the hints!
> > > >
> > > > The part that you quoted is in fact from the flume wiki and I forgot to quote it myself. But the page is quite old so I wasn't sure about the state of the feature. I thought maybe commons-configuration was used but wasn't be able to confirm it clearly (or more exactly I lost myself in this dependency, which might be used by one of my current projects somehow).
> > > >
> > > > https://cwiki.apache.org/confluence/display/FLUME/Flume+NG
> > > >
> > > > > Configuration Provider
> > > > >
> > > > > Flume NG has a pluggable configuration system called the configuration
> > > > > provider. By default, Flume NG ships with a Java property file based
> > > > > configuration system that is both simple and easy to generate
> > > > > programmatically. Flume OG has a centralized configuration system with
> > > > > a master and ZooKeeper for coordination and we recognize this is very
> > > > > appealing to some users where as others see it as overhead they simply
> > > > > don't want. We opted to make this a pluggable extension point and ship
> > > > > a basic implementation that would let many users get started quickly
> > > > > and easily. There's almost certainly enough desire for a similar
> > > > > implementation to that of Flume OG, but it isn't yet implemented.
> > > > > Users may also implement arbitrary plugins to integrate with any type
> > > > > of configuration system (JSON files, a shared RDBMS, a central system
> > > > > with ZooKeeper, and so forth). We see this as something more
> > > > > interesting to system integrators.
> > > >
> > > > I will look at the classes you pointed for my own education and I will see what I can do.
> > > >
> > > > Thanks a lot.
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB