Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop, mail # general - [DISCUSS] Spin out MR, HDFS and YARN as their own TLPs and disband Hadoop umbrella project


Copy link to this message
-
Re: [DISCUSS] Spin out MR, HDFS and YARN as their own TLPs and disband Hadoop umbrella project
Eli Collins 2012-08-31, 16:54
How about a proposal to just spin YARN off as a TLP?  Rationale:

1. YARN started as a separate project and has a more independent
community than Common/HDFS/MR (per below these communities do not
divide at sub-project boundaries) that appears to want to be even more
independent.

2. YARN is technically much easier to separate from the rest of the
code base (than separating Common and HDFS for example). Separating it
out will also help accelerate other efforts like MR2 support for
Apache Mesos.

3. It side steps a number of thorny issues (how to handle branch-1,
how to handle what Hadoop is wrt enforcing trademark, who to remove
people from the Hadoop PMC, etc) that haven't been addressed in any of
these proposals.

4. It's a proof point - if you can't make the case for YARN then
there's no way we're going to make a case for splitting the other
projects (this thread).

Ie this doesn't have to be an all-or-nothing proposition for all
sub-projects, since the communities don't fall on sub-project
boundaries.

Thanks,
Eli

On Tue, Aug 28, 2012 at 7:33 PM, Mattmann, Chris A (388J)
<[EMAIL PROTECTED]> wrote:
> [decided to minimize traffic and to simply put this in one thread]
>
> Hi Guys,
>
> See the recent discussion on these threads:
>
> YARN as its own Hadoop "sub project": http://s.apache.org/WW1
> Maintain a single committer list for the Hadoop project: http://s.apache.org/Owx
>
> ...and just pay attention to the Hadoop project over the last 3-4 years. It's operating
> as a single project, that's masking separate communities that themselves are really
> separate ASF projects.
>
> At the ASF, this has been a problem area called "umbrella" projects and over the years,
> all I've seen from them is wasted bandwidth, artificial barriers and the inventions of
> new ways to perform process mongering and to reduce the fun in developing software
> at this fantastic foundation.
>
> I've talked about umbrella projects enough. We've diverted conversation enough.
> Enough people have tried to act like there is some technical mumbo jumbo that is
> preventing the eventual act of higher power that I myself hope comes should these
> discussions prove unfruitful through normal means.
>
> *these. are. separate. projects.*
> *there.are.not.blocker.issues.from.spinning.out.these.projects.as.their.own.communities*
>
> In this email: http://s.apache.org/rSm
>
> And in the 2 subsequent follow ons in that thread, I've outlined a process that I'll copy
> through below for splitting these projects into their own TLPs:
>
> -----snip
> Process:
>
> 0. [DISCUSS] thread for <TLP name> in which you talk about #1 and #2 below, potentially draft resolution too.
>
> 1. Decide on an initial set of *PMC* members. I urge each new TLP to adopt PMC==C. See reasons I've
> already discussed.
>
> 2. Decide on a chair. Try not to VOTE for this explicitly, see if can be discussed and consensus
> can be reached (just a thought experiment). VOTE if necessary.
>
> 3. [VOTE] thread for <TLP name>
>
> 4. Create Project:
>   a. paste resolution from #0 to board@ or;
>   b. go to general@incubator and start new Incubator project.
>
> 5. infrastructure set up.
>    MLs moving; new UNIX groups; website setup;
>    SVN setup like this:
>
> svn copy -m "MR TLP." https://svn.apache.org/repos/asf/hadoop/ https://svn.apache.org/repos/asf/<insert cool MR name>; or
> svn copy -m "YARN TLP." https://svn.apache.org/repos/asf/hadoop/ https://svn.apache.org/repos/asf/<insert cool YARN name>; or
> svn copy -m "HDFS TLP." https://svn.apache.org/repos/asf/hadoop/ https://svn.apache.org/repos/asf/<insert cool HDFS name>
>
> After all 3 have been created run:
>
> svn remove -m "Remove Hadoop umbrella TLP. Split into separate projects." https://svn.apache.org/repos/asf/hadoop
>
> 6. (TLPs if 4a; Incubator podling if 4b;) proceed, collaborate, operate as distinct communities, and try to solve the code duplication/dependency
> issues from there.
>
> 7. If 4b; then graduate as TLP from Incubator.