Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Hadoop, mail # general - [DISCUSS] Spin out MR, HDFS and YARN as their own TLPs and disband Hadoop umbrella project


+
Mattmann, Chris A 2012-08-29, 02:33
+
Alejandro Abdelnur 2012-08-29, 03:50
+
Arun C Murthy 2012-08-29, 16:31
+
Tom White 2012-08-29, 17:30
+
Eric Baldeschwieler 2012-08-29, 17:42
+
Suresh Srinivas 2012-08-29, 17:02
+
Alejandro Abdelnur 2012-08-29, 17:13
+
Suresh Srinivas 2012-08-29, 17:26
+
Konstantin Boudnik 2012-08-29, 18:41
+
Arun C Murthy 2012-08-29, 17:04
+
Arun C Murthy 2012-08-29, 18:22
+
Eli Collins 2012-08-29, 18:49
+
Tom White 2012-08-29, 20:34
+
Alejandro Abdelnur 2012-08-29, 20:40
+
Todd Lipcon 2012-08-29, 21:18
+
Andrew Purtell 2012-08-30, 13:46
+
Sharad Agarwal 2012-08-30, 06:41
+
Mattmann, Chris A 2012-08-29, 23:29
+
Doug Cutting 2012-08-30, 16:17
+
Todd Lipcon 2012-08-29, 23:44
+
Konstantin Boudnik 2012-08-30, 02:59
+
Mattmann, Chris A 2012-08-29, 23:54
+
Konstantin Shvachko 2012-08-30, 10:12
+
Andrew Purtell 2012-08-30, 14:11
+
Aaron T. Myers 2012-08-30, 14:23
+
Brock Noland 2012-08-30, 14:43
+
Devaraj Das 2012-08-31, 01:28
+
Andrew Purtell 2012-08-31, 06:02
+
Mattmann, Chris A 2012-08-31, 06:15
+
Andrew Purtell 2012-08-31, 06:42
+
Mattmann, Chris A 2012-08-31, 06:50
+
Todd Lipcon 2012-08-31, 16:59
+
Alejandro Abdelnur 2012-08-31, 17:10
+
Jagane Sundar 2012-08-31, 17:24
+
Alejandro Abdelnur 2012-08-31, 17:11
+
Andrew Purtell 2012-08-31, 07:55
+
Robert Evans 2012-08-31, 14:34
+
Mattmann, Chris A 2012-08-31, 15:09
+
Doug Cutting 2012-08-31, 16:00
+
Mattmann, Chris A 2012-08-31, 16:08
+
Eric Baldeschwieler 2012-09-01, 00:23
+
Sharad Agarwal 2012-09-01, 09:59
+
Mahadev Konar 2012-08-31, 15:05
+
Mattmann, Chris A 2012-08-31, 06:36
+
Mattmann, Chris A 2012-08-30, 13:51
+
Arun C Murthy 2012-08-30, 10:25
+
Arun C Murthy 2012-08-30, 11:00
+
Todd Lipcon 2012-08-30, 00:16
+
Mattmann, Chris A 2012-08-30, 00:55
+
Mattmann, Chris A 2012-08-29, 23:34
+
Jakob Homan 2012-08-29, 21:22
+
Travis Thompson 2012-08-29, 22:30
+
Steve Loughran 2012-08-31, 11:54
+
Mattmann, Chris A 2012-08-29, 14:14
+
Robert Evans 2012-08-29, 15:17
+
Mattmann, Chris A 2012-08-29, 17:22
+
Eric Baldeschwieler 2012-08-29, 03:45
+
Michael Segel 2012-08-29, 17:26
+
Arun C Murthy 2012-08-29, 18:48
+
Mattmann, Chris A 2012-08-29, 23:20
+
Eli Collins 2012-08-30, 05:46
+
Arun C Murthy 2012-08-30, 06:31
+
Eli Collins 2012-08-30, 07:02
+
Alejandro Abdelnur 2012-08-30, 07:11
+
Arun C Murthy 2012-08-30, 12:29
+
Todd Lipcon 2012-08-29, 23:35
+
Aaron T. Myers 2012-08-29, 23:53
+
Konstantin Boudnik 2012-08-29, 23:47
+
Todd Lipcon 2012-08-29, 23:48
+
Arun C Murthy 2012-08-30, 01:52
+
Inder.dev Java 2012-08-30, 16:33
+
Doug Cutting 2012-08-30, 17:00
+
Owen OMalley 2012-08-30, 18:25
+
Inder.dev Java 2012-08-31, 19:00
+
Doug Cutting 2012-08-31, 20:44
+
Eric Baldeschwieler 2012-08-31, 22:43
+
Eli Collins 2012-08-29, 18:41
+
Mattmann, Chris A 2012-08-29, 23:19
+
Eli Collins 2012-08-30, 05:38
+
Mattmann, Chris A 2012-08-30, 06:06
+
Eli Collins 2012-08-30, 06:18
+
Mattmann, Chris A 2012-08-30, 06:31
+
Eli Collins 2012-08-30, 07:17
+
Konstantin Boudnik 2012-08-29, 23:27
+
Mattmann, Chris A 2012-08-29, 23:32
+
Arun C Murthy 2012-08-30, 01:47
+
Konstantin Boudnik 2012-08-29, 23:40
+
Jun Ping Du 2012-08-29, 18:35
+
Chris Douglas 2012-08-31, 01:24
+
Vinod Kumar Vavilapalli 2012-08-31, 03:35
+
Roman Shaposhnik 2012-08-31, 15:59
+
Andrew Purtell 2012-09-01, 13:21
+
Eli Collins 2012-08-31, 16:54
+
Robert Evans 2012-08-31, 16:58
+
Todd Lipcon 2012-08-31, 17:06
+
Robert Evans 2012-08-31, 18:15
Copy link to this message
-
Re: [DISCUSS] Spin out MR, HDFS and YARN as their own TLPs and disband Hadoop umbrella project
Andrew Purtell 2012-09-01, 13:32
Or resurrect MR(v1) in Apache Hadoop as Apache YARN becomes a TLP, and let
the new YARN TLP decide if they want to use the Hadoop MR artifacts and/or
contribute patches that harmonize the implementation with theirs, or pursue
an alternate MR implementation within their larger framework.

I'd imagine such a MR(v1) in Hadoop, if this happened, would concentrate on
performance improvements, maybe such things as alternate shuffle plugins.
Perhaps a HA JobTracker for parity with HDFS. But we could expect a clear
separation where next generation framework work would be continued in and
centered upon YARN, while Hadoop remains... well, Hadoop.

On Friday, August 31, 2012, Robert Evans wrote:

> The problem there is that YARN depends on Common, and MapReduce depends on
> YARN, so we would either have a circular dependency or we would have to
> split off MapRedcue too.
>
> --Bobby
>
> On 8/31/12 11:54 AM, "Eli Collins" <[EMAIL PROTECTED]> wrote:
>
> >How about a proposal to just spin YARN off as a TLP?  Rationale:
> >
> >1. YARN started as a separate project and has a more independent
> >community than Common/HDFS/MR (per below these communities do not
> >divide at sub-project boundaries) that appears to want to be even more
> >independent.
> >
> >2. YARN is technically much easier to separate from the rest of the
> >code base (than separating Common and HDFS for example). Separating it
> >out will also help accelerate other efforts like MR2 support for
> >Apache Mesos.
> >
> >3. It side steps a number of thorny issues (how to handle branch-1,
> >how to handle what Hadoop is wrt enforcing trademark, who to remove
> >people from the Hadoop PMC, etc) that haven't been addressed in any of
> >these proposals.
> >
> >4. It's a proof point - if you can't make the case for YARN then
> >there's no way we're going to make a case for splitting the other
> >projects (this thread).
> >
> >Ie this doesn't have to be an all-or-nothing proposition for all
> >sub-projects, since the communities don't fall on sub-project
> >boundaries.
> >
> >Thanks,
> >Eli
> >
> >On Tue, Aug 28, 2012 at 7:33 PM, Mattmann, Chris A (388J)
> ><[EMAIL PROTECTED]> wrote:
> >> [decided to minimize traffic and to simply put this in one thread]
> >>
> >> Hi Guys,
> >>
> >> See the recent discussion on these threads:
> >>
> >> YARN as its own Hadoop "sub project": http://s.apache.org/WW1
> >> Maintain a single committer list for the Hadoop project:
> >>http://s.apache.org/Owx
> >>
> >> ...and just pay attention to the Hadoop project over the last 3-4
> >>years. It's operating
> >> as a single project, that's masking separate communities that
> >>themselves are really
> >> separate ASF projects.
> >>
> >> At the ASF, this has been a problem area called "umbrella" projects and
> >>over the years,
> >> all I've seen from them is wasted bandwidth, artificial barriers and
> >>the inventions of
> >> new ways to perform process mongering and to reduce the fun in
> >>developing software
> >> at this fantastic foundation.
> >>
> >> I've talked about umbrella projects enough. We've diverted conversation
> >>enough.
> >> Enough people have tried to act like there is some technical mumbo
> >>jumbo that is
> >> preventing the eventual act of higher power that I myself hope comes
> >>should these
> >> discussions prove unfruitful through normal means.
> >>
> >> *these. are. separate. projects.*
> >>
> >>*there.are.not.blocker.issues.from.spinning.out.these.projects.as.their.o
> >>wn.communities*
> >>
> >> In this email: http://s.apache.org/rSm
> >>
> >> And in the 2 subsequent follow ons in that thread, I've outlined a
> >>process that I'll copy
> >> through below for splitting these projects into their own TLPs:
> >>
> >> -----snip
> >> Process:
> >>
> >> 0. [DISCUSS] thread for <TLP name> in which you talk about #1 and #2
> >>below, potentially draft resolution too.
> >>
> >> 1. Decide on an initial set of *PMC* members. I urge each new TLP to
> >>adopt PMC==C. See reasons I've

Best regards,

   - Andy

Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)
+
Arun C Murthy 2012-09-03, 11:02