Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # dev >> Regarding the adding of additional sinks/sources for various DB's

Copy link to this message
Re: Regarding the adding of additional sinks/sources for various DB's
 I guess my questions is why the suggestion to use the elastic search
model, is there something you see that's not working?
On Mon, Nov 25, 2013 at 5:34 PM, Israel Ekpo <[EMAIL PROTECTED]> wrote:

> I think we can take a page of out the ElasticSearch playbook.
> http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-plugins.html
> I like the model they follow.
> The Flume architecture makes it easy for plugins at any layer (source,
> interceptor, sink etc)
> Contributors can host plugins on github and manage the documentation and
> maintenance of the plugin.
> Others can chip it when possible to improve or maintain the plugins.
> This will still allow new features to the project without necessarily
> meaning that Flume committers are on the hook for maintaining it.
> *Author and Instructor for the Upcoming Book and Lecture Series*
> *Massive Log Data Aggregation, Processing, Searching and Visualization with
> Open Source Software*
> *http://massivelogdata.com <http://massivelogdata.com>*
> On Mon, Nov 25, 2013 at 3:14 AM, Juhani Connolly <
> > Hey guys,
> >
> > What I write here is all just my personal opinion and I'm writing in
> hopes
> > of starting a discussion and/or getting feedback. I know I've not been
> very
> > active on the project recently(due to other engagements) but do still
> want
> > it to succeed and hope to find more time for it eventually.
> >
> > Right now I see new/active issues for the addition of Redis and Kafka
> > sinks, and while they're nice features, I'm personally concerned about
> > feature bloat of the project. There are dozens of interceptors, sinks and
> > sources that can be thought of, but most of them are very specific to a
> > specific use-case.
> >
> > Every time we add a new component we're also committing to maintaining it
> > over future releases, even if the original contributor gets too busy for
> > it. The more such components get added, the more we will get distracted
> > from improving core features and getting rid of issues affecting them.
> >
> > For these reasons I generally haven't submitted components we developed
> > for internal use(because they are too specific to our use cases), just
> > passing back fixes that fix bugs or apply to the core project.
> >
> > For these reasons I think we may want to consider either a) being more
> > selective regarding additional component submissions or b) make a contrib
> > directory to the project which includes the components but doesn't
> > guarrantee ongoing maintenance or compatibility.
> >
> > On the flip side of course, taking approaches like this may discourage
> new
> > contributors and could thus be considered a negative, and if many people
> > feel this way they should definitely share their thoughts.
> >
> > I'd be curious to know what others think, and what direction they hope to
> > take the project in the future.
> >