Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume, mail # dev - [DISCUSS] Feature bloat and contrib module

Copy link to this message
Re: [DISCUSS] Feature bloat and contrib module
Ashish 2013-12-17, 05:17
@Israel - IMHO JIRA is not a good use-case for these discussion.
The discussion can easily be tracked on flume.markmail.com and the link is
On Tue, Dec 17, 2013 at 7:19 AM, Mike Percy <[EMAIL PROTECTED]> wrote:

> > I have created a JIRA Brainstorming task to track this.
> Hmm, I think there is a risk of losing this discussion in the flood of JIRA
> traffic due to email filters. So I'm going to respond here on this thread.
> If you want to reference this thread in the future, you can use this URL:
> http://markmail.org/message/7x7tewbxqw4ubjyp
> My 2 cents below.
> > Do contrib components get released with Flume? Can they break
> compatibility with older versions (what does this mean if they are not
> getting released?) etc. How supported are these?
> I think you hit the nail on the head here.
> Recently I commented on the Storm sink indicating I would not help maintain
> it therefore I would not help review it. Implicitly, I meant that someone
> else should probably take responsibility for maintaining it to some extent.
> So I think having at least one person who hopefully has the bandwidth to
> understand and maintain a module should be a prerequisite for inclusion
> into the main line of Flume (i.e. the bits included in a typical Flume
> release).

Mike - You have echoed my thoughts.

We have the same situation in MINA (and in Netty as well). People write a
lot of codecs, but we can't get
all of them into the release. A Codec would be brought in only if one of
the Committers volunteers to maintain it.

This was my thought for Flume as well, if one of the committer volunteers
to maintain it, bring it in, else we have to find
a way to keep things easy for Users. How, not sure at the moment.

> Generally, my concern with creating a contrib module is that, in my view,
> it is where code goes to die, since why have a contrib module unless we
> have no intention of maintaining that code? As an example, see the various
> states of stability of code in Pig's piggybank. Is there good stuff in
> there? Absolutely. Is everything in there relatively stable / usable, even
> on a release? Nope. Those things may or may not work at any time.
> Personally I'm not sure adding a contrib is a very good idea.

So let's assume, we have contrib, and if it gets release with Flume core,
we don't have a choice but to atleast ensure
it's compiling and tests are running. Sooner or later, one of the core jar
upgrade will force us to fix the code in contrib.
People can contribute patches, but this would still consume Dev cycles.

If contrib is source only release or not compatible with current release,
it better not be part of released.
This will add a lot of confusion for users.
Zookeeper does have contrib, not sure how they feel about it

Still, I am worried about impact it might have on future contributions
> As another dimension to this discussion, I think there is a limit to the
> number of dependencies Flume can reasonably pull in and keep straight
> without shading or classloading tricks, which themselves add another layer
> of pain/difficulty to debugging.
> > https://cwiki.apache.org/confluence/display/FLUME/Flume+NG+Plugins
> I think this page has a lot of value but maybe the problem with it is that
> it is not accessible from the home page of http://flume.apache.org/
> TL;DR:
> 1. In my view we should only accept new, big components in the main project
> if they have a good chance of continuing to be maintained.

+1, if one of the Committers volunteers to maintain it
> 2. The more dependencies Flume has, the harder it is to upgrade any one
> component and keep all the dependencies from conflicting (i.e. JAR hell).

We are almost there :)
> 3. Is there an alternative to contrib that would solve the problem of
> giving new plugins a home if they cannot be included in the main project?
> Is there a way we could make plugins that live on e.g. GitHub more visible?
> 4. If we add a contrib module, I agree that we need to be clear on why

Summarizing my suggestions
1. For a contribution to make it into Flume, one of committers had to step
2. Not very much convinced with contrib, but don't have solution otherwise
as well

Blog: http://www.ashishpaliwal.com/blog
My Photo Galleries: http://www.pbase.com/ashishpaliwal