Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # dev >> [DISCUSS] Feature bloat and contrib module


Copy link to this message
-
Re: [DISCUSS] Feature bloat and contrib module
One of the things that OS distros tend to do is build all the common deps
for things that tend to conflict, and in our case I don't think even BigTop
is planning on building Guava, Protobuf, or other commonly-conflicting
libraries. So I am still of the opinion that classloader isolation is
needed if we plan to continue expanding scope and including more
off-the-shelf integrations in the core Flume project.

Another option is to use maven-shade on individual components, that might
work for many cases but some projects themselves (notably Flume) hard-code
class names and don't work with shading. Granted, usually that is fixable
provided the cycles are put in to work around the issues that come up when
shading the upstream components.

Mike
On Thu, Jan 16, 2014 at 12:44 AM, Ashish <[EMAIL PROTECTED]> wrote:

> Folks,
>
> Can we close this thread, one way or the other. It's been a while.
>
> thanks
> ashish
>
>
> On Mon, Dec 23, 2013 at 10:34 PM, Gabriel Commeau <
> [EMAIL PROTECTED]
> > wrote:
>
> > Hi,
> >
> > IMHO, contrib modules seem better for the following reasons:
> > 1. Keep the core as thin as possible. I like the idea of a pluggeable
> Flume
> > where the user adds the components needed, and only these. I imagine that
> > realistically, most users only use a handful of components, and therefore
> > don't need the whole library of every existing (or supported) sink and
> > source on their localhost. We could make the process of adding/removing
> > components easier, so that it becomes trivial for the user to
> > download/install/activate them.
> > 2. License considerations. I can envision cases where one would want to
> > integrate Flume with another system that uses a license that's not
> > compatible with Apache's. So whether a contributor needs or wants to use
> a
> > different license, this contribution cannot currently be added to Flume.
> > I'm
> > not an expert on licenses, but I wonder if it would be possible to
> include
> > these contributions using a contrib module.
> > 3. Easiest way in. "Getting in" becomes trivial and open to all. Seems to
> > me
> > like the best way to grow the project.
> > 4. Community-based out. With a contrib project, we actually don't really
> > need to move contributions out. The community, if able to vote or report
> > usage naturally manages which contributions are used and which aren't.
> > 5. Competition and maintenance. As software engineers, there are always
> > tradeoffs we need to make. Imagine a component that could have its
> > performances increased at the cost of, for example, compatibility with
> some
> > other systems. Why would this optimization have to conflict with Apache's
> > main component? Couldn't both live side-by-side, and let the user choose
> > the
> > one that better fit his/her specific context and requirements?
> > So to answer the original discussion questions: I'd argue that contrib
> > modules would benefit Flume, that they should be released on their own
> > schedule, supported independently, and be compatible with whatever
> version
> > of Flume the authors wish.
> > I like cathedrals, and I tend to design my applications like that. But in
> > this case, I believe a little bit of bazaar would be best.
> > I hope this helps.
> >
> > Gabriel
> >
> > From:  Bruno Mahé <[EMAIL PROTECTED]>
> > Reply-To:  <[EMAIL PROTECTED]>
> > Date:  Saturday, December 21, 2013 4:29 PM
> > To:  <[EMAIL PROTECTED]>
> > Subject:  Re: [DISCUSS] Feature bloat and contrib module
> >
> > See inline.
> >
> > On 12/20/2013 04:01 PM, Mike Percy wrote:
> > >  On Mon, Dec 16, 2013 at 11:34 PM, Bruno Mahé <[EMAIL PROTECTED]>
> wrote:
> > >>
> > >>  Summarizing my suggestions:
> > >>  * Commiters are not the sole developers. There is no reason for
> > commiters
> > >>  to take all these responsibilities on their shoulders. Also developer
> > !> > >>  commiter.
> > >>  * Easy IN, Easy OUT. If no one volunteers to maintain something, then
> > >>  there is no reason to keep it since the community is not interested