Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # dev >> [DISCUSS] Feature bloat and contrib module


Copy link to this message
-
Re: [DISCUSS] Feature bloat and contrib module
> I have created a JIRA Brainstorming task to track this.

Hmm, I think there is a risk of losing this discussion in the flood of JIRA
traffic due to email filters. So I'm going to respond here on this thread.
If you want to reference this thread in the future, you can use this URL:
http://markmail.org/message/7x7tewbxqw4ubjyp

My 2 cents below.

> Do contrib components get released with Flume? Can they break
compatibility with older versions (what does this mean if they are not
getting released?) etc. How supported are these?

I think you hit the nail on the head here.

Recently I commented on the Storm sink indicating I would not help maintain
it therefore I would not help review it. Implicitly, I meant that someone
else should probably take responsibility for maintaining it to some extent.
So I think having at least one person who hopefully has the bandwidth to
understand and maintain a module should be a prerequisite for inclusion
into the main line of Flume (i.e. the bits included in a typical Flume
release).

Generally, my concern with creating a contrib module is that, in my view,
it is where code goes to die, since why have a contrib module unless we
have no intention of maintaining that code? As an example, see the various
states of stability of code in Pig's piggybank. Is there good stuff in
there? Absolutely. Is everything in there relatively stable / usable, even
on a release? Nope. Those things may or may not work at any time.
Personally I'm not sure adding a contrib is a very good idea.

As another dimension to this discussion, I think there is a limit to the
number of dependencies Flume can reasonably pull in and keep straight
without shading or classloading tricks, which themselves add another layer
of pain/difficulty to debugging.

> https://cwiki.apache.org/confluence/display/FLUME/Flume+NG+Plugins

I think this page has a lot of value but maybe the problem with it is that
it is not accessible from the home page of http://flume.apache.org/

TL;DR:
1. In my view we should only accept new, big components in the main project
if they have a good chance of continuing to be maintained.
2. The more dependencies Flume has, the harder it is to upgrade any one
component and keep all the dependencies from conflicting (i.e. JAR hell).
3. Is there an alternative to contrib that would solve the problem of
giving new plugins a home if they cannot be included in the main project?
Is there a way we could make plugins that live on e.g. GitHub more visible?
4. If we add a contrib module, I agree that we need to be clear on why
contrib exists and what it means to have something live in there.

Mike
On Mon, Dec 16, 2013 at 12:17 PM, Israel Ekpo <[EMAIL PROTECTED]> wrote:

> I have created a JIRA Brainstorming task to track this.
>
> I think it will be very helpful for us to collect our thoughts on the JIRA
> ticket for future references.
>
> I think this is a very important topic.
>
> https://issues.apache.org/jira/browse/FLUME-2274
>
>
>
>
> On Mon, Dec 16, 2013 at 12:51 PM, Ashish <[EMAIL PROTECTED]> wrote:
>
> > Quick question before I put down my thoughts. What does "core Flume"
> > represents? Is it the current Flume trunk structure or just few modules?
> >
> >
> > On Sat, Dec 14, 2013 at 5:51 AM, Hari Shreedharan <
> > [EMAIL PROTECTED]
> > > wrote:
> >
> > > Hi all
> > >
> > > Over the past few weeks, there were some discussions about including
> > > additional features in core flume itself. This led to a discussion
> about
> > > adding a contrib module. I thought I’d start an official discussion
> > > regarding this. The argument for contrib module was that there are too
> > many
> > > components without generic use which are getting submitted/committed.
> The
> > > use-cases for such components  are limited, and hence they should not
> be
> > > part of core flume itself. First, we should answer the question if we
> > want
> > > to separate components into a contrib module and why? What components
> go
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB