Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # dev >> Regarding the adding of additional sinks/sources for various DB's


Copy link to this message
-
Re: Regarding the adding of additional sinks/sources for various DB's
Juhani and others,

I agree that it does make sense to add a contrib module to flume where
non-hadoopy stuff can go. I will start a discussion on this early next week.

Hari

On Thursday, November 28, 2013, Steve Morin wrote:

> Israel,
>  I guess my questions is why the suggestion to use the elastic search
> model, is there something you see that's not working?
> -Steve
>
>
> On Mon, Nov 25, 2013 at 5:34 PM, Israel Ekpo <[EMAIL PROTECTED]<javascript:;>>
> wrote:
>
> > I think we can take a page of out the ElasticSearch playbook.
> >
> >
> >
> http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-plugins.html
> >
> > I like the model they follow.
> >
> > The Flume architecture makes it easy for plugins at any layer (source,
> > interceptor, sink etc)
> >
> > Contributors can host plugins on github and manage the documentation and
> > maintenance of the plugin.
> >
> > Others can chip it when possible to improve or maintain the plugins.
> >
> > This will still allow new features to the project without necessarily
> > meaning that Flume committers are on the hook for maintaining it.
> >
> >
> >
> > *Author and Instructor for the Upcoming Book and Lecture Series*
> > *Massive Log Data Aggregation, Processing, Searching and Visualization
> with
> > Open Source Software*
> > *http://massivelogdata.com <http://massivelogdata.com>*
> >
> >
> > On Mon, Nov 25, 2013 at 3:14 AM, Juhani Connolly <
> > [EMAIL PROTECTED] <javascript:;>> wrote:
> >
> > > Hey guys,
> > >
> > > What I write here is all just my personal opinion and I'm writing in
> > hopes
> > > of starting a discussion and/or getting feedback. I know I've not been
> > very
> > > active on the project recently(due to other engagements) but do still
> > want
> > > it to succeed and hope to find more time for it eventually.
> > >
> > > Right now I see new/active issues for the addition of Redis and Kafka
> > > sinks, and while they're nice features, I'm personally concerned about
> > > feature bloat of the project. There are dozens of interceptors, sinks
> and
> > > sources that can be thought of, but most of them are very specific to a
> > > specific use-case.
> > >
> > > Every time we add a new component we're also committing to maintaining
> it
> > > over future releases, even if the original contributor gets too busy
> for
> > > it. The more such components get added, the more we will get distracted
> > > from improving core features and getting rid of issues affecting them.
> > >
> > > For these reasons I generally haven't submitted components we developed
> > > for internal use(because they are too specific to our use cases), just
> > > passing back fixes that fix bugs or apply to the core project.
> > >
> > > For these reasons I think we may want to consider either a) being more
> > > selective regarding additional component submissions or b) make a
> contrib
> > > directory to the project which includes the components but doesn't
> > > guarrantee ongoing maintenance or compatibility.
> > >
> > > On the flip side of course, taking approaches like this may discourage
> > new
> > > contributors and could thus be considered a negative, and if many
> people
> > > feel this way they should definitely share their thoughts.
> > >
> > > I'd be curious to know what others think, and what direction they hope
> to
> > > take the project in the future.
> > >
> >
>