Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Hadoop, mail # general - [DISCUSS] Spin out MR, HDFS and YARN as their own TLPs and disband Hadoop umbrella project


+
Mattmann, Chris A 2012-08-29, 02:33
Copy link to this message
-
Re: [DISCUSS] Spin out MR, HDFS and YARN as their own TLPs and disband Hadoop umbrella project
Alejandro Abdelnur 2012-08-29, 03:50
Chris, thanks for initiating the discussion.

IMO a pre-requisite to this is to figure out how we'll handle the following:

* Where does common stuff lives?
* What are the public interfaces of each project (towards the other projects)?
* How do we do development/releases? In tandem? Separate? How this
will work in practice, currently we are constantly tweaking things
inter-projects, sometimes in the same JIRAs, sometimes in follow up
JIRAs.

Thoughts?

Thxs.

On Tue, Aug 28, 2012 at 7:33 PM, Mattmann, Chris A (388J)
<[EMAIL PROTECTED]> wrote:
> [decided to minimize traffic and to simply put this in one thread]
>
> Hi Guys,
>
> See the recent discussion on these threads:
>
> YARN as its own Hadoop "sub project": http://s.apache.org/WW1
> Maintain a single committer list for the Hadoop project: http://s.apache.org/Owx
>
> ...and just pay attention to the Hadoop project over the last 3-4 years. It's operating
> as a single project, that's masking separate communities that themselves are really
> separate ASF projects.
>
> At the ASF, this has been a problem area called "umbrella" projects and over the years,
> all I've seen from them is wasted bandwidth, artificial barriers and the inventions of
> new ways to perform process mongering and to reduce the fun in developing software
> at this fantastic foundation.
>
> I've talked about umbrella projects enough. We've diverted conversation enough.
> Enough people have tried to act like there is some technical mumbo jumbo that is
> preventing the eventual act of higher power that I myself hope comes should these
> discussions prove unfruitful through normal means.
>
> *these. are. separate. projects.*
> *there.are.not.blocker.issues.from.spinning.out.these.projects.as.their.own.communities*
>
> In this email: http://s.apache.org/rSm
>
> And in the 2 subsequent follow ons in that thread, I've outlined a process that I'll copy
> through below for splitting these projects into their own TLPs:
>
> -----snip
> Process:
>
> 0. [DISCUSS] thread for <TLP name> in which you talk about #1 and #2 below, potentially draft resolution too.
>
> 1. Decide on an initial set of *PMC* members. I urge each new TLP to adopt PMC==C. See reasons I've
> already discussed.
>
> 2. Decide on a chair. Try not to VOTE for this explicitly, see if can be discussed and consensus
> can be reached (just a thought experiment). VOTE if necessary.
>
> 3. [VOTE] thread for <TLP name>
>
> 4. Create Project:
>   a. paste resolution from #0 to board@ or;
>   b. go to general@incubator and start new Incubator project.
>
> 5. infrastructure set up.
>    MLs moving; new UNIX groups; website setup;
>    SVN setup like this:
>
> svn copy -m "MR TLP." https://svn.apache.org/repos/asf/hadoop/ https://svn.apache.org/repos/asf/<insert cool MR name>; or
> svn copy -m "YARN TLP." https://svn.apache.org/repos/asf/hadoop/ https://svn.apache.org/repos/asf/<insert cool YARN name>; or
> svn copy -m "HDFS TLP." https://svn.apache.org/repos/asf/hadoop/ https://svn.apache.org/repos/asf/<insert cool HDFS name>
>
> After all 3 have been created run:
>
> svn remove -m "Remove Hadoop umbrella TLP. Split into separate projects." https://svn.apache.org/repos/asf/hadoop
>
> 6. (TLPs if 4a; Incubator podling if 4b;) proceed, collaborate, operate as distinct communities, and try to solve the code duplication/dependency
> issues from there.
>
> 7. If 4b; then graduate as TLP from Incubator.
>
> -----snip
>
> So that's my proposal.
>
> Thanks guys.
>
> Cheers,
> Chris
>
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> Chris Mattmann, Ph.D.
> Senior Computer Scientist
> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> Office: 171-266B, Mailstop: 171-246
> Email: [EMAIL PROTECTED]
> WWW:   http://sunset.usc.edu/~mattmann/
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> Adjunct Assistant Professor, Computer Science Department
> University of Southern California, Los Angeles, CA 90089 USA

Alejandro
+
Arun C Murthy 2012-08-29, 16:31
+
Tom White 2012-08-29, 17:30
+
Eric Baldeschwieler 2012-08-29, 17:42
+
Suresh Srinivas 2012-08-29, 17:02
+
Alejandro Abdelnur 2012-08-29, 17:13
+
Suresh Srinivas 2012-08-29, 17:26
+
Konstantin Boudnik 2012-08-29, 18:41
+
Arun C Murthy 2012-08-29, 17:04
+
Arun C Murthy 2012-08-29, 18:22
+
Eli Collins 2012-08-29, 18:49
+
Tom White 2012-08-29, 20:34
+
Alejandro Abdelnur 2012-08-29, 20:40
+
Todd Lipcon 2012-08-29, 21:18
+
Andrew Purtell 2012-08-30, 13:46
+
Sharad Agarwal 2012-08-30, 06:41
+
Mattmann, Chris A 2012-08-29, 23:29
+
Doug Cutting 2012-08-30, 16:17
+
Todd Lipcon 2012-08-29, 23:44
+
Konstantin Boudnik 2012-08-30, 02:59
+
Mattmann, Chris A 2012-08-29, 23:54
+
Konstantin Shvachko 2012-08-30, 10:12
+
Andrew Purtell 2012-08-30, 14:11
+
Aaron T. Myers 2012-08-30, 14:23
+
Brock Noland 2012-08-30, 14:43
+
Devaraj Das 2012-08-31, 01:28
+
Andrew Purtell 2012-08-31, 06:02
+
Mattmann, Chris A 2012-08-31, 06:15
+
Andrew Purtell 2012-08-31, 06:42
+
Mattmann, Chris A 2012-08-31, 06:50
+
Todd Lipcon 2012-08-31, 16:59
+
Alejandro Abdelnur 2012-08-31, 17:10
+
Jagane Sundar 2012-08-31, 17:24
+
Alejandro Abdelnur 2012-08-31, 17:11
+
Andrew Purtell 2012-08-31, 07:55
+
Robert Evans 2012-08-31, 14:34
+
Mattmann, Chris A 2012-08-31, 15:09
+
Doug Cutting 2012-08-31, 16:00
+
Mattmann, Chris A 2012-08-31, 16:08
+
Eric Baldeschwieler 2012-09-01, 00:23
+
Sharad Agarwal 2012-09-01, 09:59
+
Mahadev Konar 2012-08-31, 15:05
+
Mattmann, Chris A 2012-08-31, 06:36
+
Mattmann, Chris A 2012-08-30, 13:51
+
Arun C Murthy 2012-08-30, 10:25
+
Arun C Murthy 2012-08-30, 11:00
+
Todd Lipcon 2012-08-30, 00:16
+
Mattmann, Chris A 2012-08-30, 00:55
+
Mattmann, Chris A 2012-08-29, 23:34
+
Jakob Homan 2012-08-29, 21:22
+
Travis Thompson 2012-08-29, 22:30
+
Steve Loughran 2012-08-31, 11:54
+
Mattmann, Chris A 2012-08-29, 14:14
+
Robert Evans 2012-08-29, 15:17
+
Mattmann, Chris A 2012-08-29, 17:22
+
Eric Baldeschwieler 2012-08-29, 03:45
+
Michael Segel 2012-08-29, 17:26
+
Arun C Murthy 2012-08-29, 18:48
+
Mattmann, Chris A 2012-08-29, 23:20
+
Eli Collins 2012-08-30, 05:46
+
Arun C Murthy 2012-08-30, 06:31
+
Eli Collins 2012-08-30, 07:02
+
Alejandro Abdelnur 2012-08-30, 07:11
+
Arun C Murthy 2012-08-30, 12:29
+
Todd Lipcon 2012-08-29, 23:35
+
Aaron T. Myers 2012-08-29, 23:53
+
Konstantin Boudnik 2012-08-29, 23:47
+
Todd Lipcon 2012-08-29, 23:48
+
Arun C Murthy 2012-08-30, 01:52
+
Inder.dev Java 2012-08-30, 16:33
+
Doug Cutting 2012-08-30, 17:00
+
Owen OMalley 2012-08-30, 18:25
+
Inder.dev Java 2012-08-31, 19:00
+
Doug Cutting 2012-08-31, 20:44
+
Eric Baldeschwieler 2012-08-31, 22:43
+
Eli Collins 2012-08-29, 18:41
+
Mattmann, Chris A 2012-08-29, 23:19
+
Eli Collins 2012-08-30, 05:38
+
Mattmann, Chris A 2012-08-30, 06:06
+
Eli Collins 2012-08-30, 06:18
+
Mattmann, Chris A 2012-08-30, 06:31
+
Eli Collins 2012-08-30, 07:17
+
Konstantin Boudnik 2012-08-29, 23:27
+
Mattmann, Chris A 2012-08-29, 23:32
+
Arun C Murthy 2012-08-30, 01:47
+
Konstantin Boudnik 2012-08-29, 23:40
+
Jun Ping Du 2012-08-29, 18:35
+
Chris Douglas 2012-08-31, 01:24
+
Vinod Kumar Vavilapalli 2012-08-31, 03:35
+
Roman Shaposhnik 2012-08-31, 15:59
+
Andrew Purtell 2012-09-01, 13:21
+
Eli Collins 2012-08-31, 16:54
+
Robert Evans 2012-08-31, 16:58
+
Todd Lipcon 2012-08-31, 17:06
+
Robert Evans 2012-08-31, 18:15
+
Andrew Purtell 2012-09-01, 13:32
+
Arun C Murthy 2012-09-03, 11:02