Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # dev >> Regarding the adding of additional sinks/sources for various DB's


Copy link to this message
-
Re: Regarding the adding of additional sinks/sources for various DB's
Israel,
 I guess my questions is why the suggestion to use the elastic search
model, is there something you see that's not working?
-Steve
On Mon, Nov 25, 2013 at 5:34 PM, Israel Ekpo <[EMAIL PROTECTED]> wrote:

> I think we can take a page of out the ElasticSearch playbook.
>
>
> http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-plugins.html
>
> I like the model they follow.
>
> The Flume architecture makes it easy for plugins at any layer (source,
> interceptor, sink etc)
>
> Contributors can host plugins on github and manage the documentation and
> maintenance of the plugin.
>
> Others can chip it when possible to improve or maintain the plugins.
>
> This will still allow new features to the project without necessarily
> meaning that Flume committers are on the hook for maintaining it.
>
>
>
> *Author and Instructor for the Upcoming Book and Lecture Series*
> *Massive Log Data Aggregation, Processing, Searching and Visualization with
> Open Source Software*
> *http://massivelogdata.com <http://massivelogdata.com>*
>
>
> On Mon, Nov 25, 2013 at 3:14 AM, Juhani Connolly <
> [EMAIL PROTECTED]> wrote:
>
> > Hey guys,
> >
> > What I write here is all just my personal opinion and I'm writing in
> hopes
> > of starting a discussion and/or getting feedback. I know I've not been
> very
> > active on the project recently(due to other engagements) but do still
> want
> > it to succeed and hope to find more time for it eventually.
> >
> > Right now I see new/active issues for the addition of Redis and Kafka
> > sinks, and while they're nice features, I'm personally concerned about
> > feature bloat of the project. There are dozens of interceptors, sinks and
> > sources that can be thought of, but most of them are very specific to a
> > specific use-case.
> >
> > Every time we add a new component we're also committing to maintaining it
> > over future releases, even if the original contributor gets too busy for
> > it. The more such components get added, the more we will get distracted
> > from improving core features and getting rid of issues affecting them.
> >
> > For these reasons I generally haven't submitted components we developed
> > for internal use(because they are too specific to our use cases), just
> > passing back fixes that fix bugs or apply to the core project.
> >
> > For these reasons I think we may want to consider either a) being more
> > selective regarding additional component submissions or b) make a contrib
> > directory to the project which includes the components but doesn't
> > guarrantee ongoing maintenance or compatibility.
> >
> > On the flip side of course, taking approaches like this may discourage
> new
> > contributors and could thus be considered a negative, and if many people
> > feel this way they should definitely share their thoughts.
> >
> > I'd be curious to know what others think, and what direction they hope to
> > take the project in the future.
> >
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB