Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Hadoop >> mail # general >> [DISCUSS] Spin out MR, HDFS and YARN as their own TLPs and disband Hadoop umbrella project


+
Mattmann, Chris A 2012-08-29, 02:33
+
Alejandro Abdelnur 2012-08-29, 03:50
+
Arun C Murthy 2012-08-29, 16:31
+
Tom White 2012-08-29, 17:30
+
Eric Baldeschwieler 2012-08-29, 17:42
Copy link to this message
-
Re: [DISCUSS] Spin out MR, HDFS and YARN as their own TLPs and disband Hadoop umbrella project
I am +1 for splitting up the projects. This is the step in the right
direction. There will be challenges along the way. I am confident we can
solve them.

Robert and Alejandro have brought up good questions. Here are my thoughts:
- For first one or two releases all the projects can coordinate and do the
releases together. This should help simplify the immediate work needed.
This should also help in us meeting the release timelines that we are
working towards. As the split makes progress, this cross project
coordination will no longer be necessary. I volunteer to RM these releases
and do the needed co-ordination from HDFS.
- As regards to APIs, currently we have LimitedPrivate APIs for related
projects. This has been used by HBase as well. We need to think about a
timeline by when we can mark these APIs stable. They should remain
LimitedPrivate. Any rare changes to APIs requires only co-ordination among
the projects and no user applications (which we have not control over) is
affected.
- I agree with Arun that the common can move with HDFS.

Regards,
Suresh

On Wed, Aug 29, 2012 at 9:31 AM, Arun C Murthy <[EMAIL PROTECTED]> wrote:

>
> On Aug 28, 2012, at 8:50 PM, Alejandro Abdelnur wrote:
>
> > Chris, thanks for initiating the discussion.
>
> Likewise, thanks Chris!
>
> >
> > IMO a pre-requisite to this is to figure out how we'll handle the
> following:
> >
>
>
> Good points - I'd recommend we keep Common and HDFS in the same project.
> Yes, MR/YARN will need some changes in Common occasionally, but core pieces
> like RPC have been maintained by HDFS folks over time anyway e.g. move to
> ProtoBufs were led by Sanjay, Suresh, Todd, Jitendra et al.
>
> We can move SequenceFile into MR if necessary and keep same package names
> for compatibility.
>
> We should, of course, stop tweaking things in different projects in the
> same jira - we've been reasonably good at not doing that.
>
> Thoughts?
>
> Arun
>
> > * Where does common stuff lives?
> > * What are the public interfaces of each project (towards the other
> projects)?
> > * How do we do development/releases? In tandem? Separate? How this
> > will work in practice, currently we are constantly tweaking things
> > inter-projects, sometimes in the same JIRAs, sometimes in follow up
> > JIRAs.
> >
> > Thoughts?
> >
> > Thxs.
> >
> > On Tue, Aug 28, 2012 at 7:33 PM, Mattmann, Chris A (388J)
> > <[EMAIL PROTECTED]> wrote:
> >> [decided to minimize traffic and to simply put this in one thread]
> >>
> >> Hi Guys,
> >>
> >> See the recent discussion on these threads:
> >>
> >> YARN as its own Hadoop "sub project": http://s.apache.org/WW1
> >> Maintain a single committer list for the Hadoop project:
> http://s.apache.org/Owx
> >>
> >> ...and just pay attention to the Hadoop project over the last 3-4
> years. It's operating
> >> as a single project, that's masking separate communities that
> themselves are really
> >> separate ASF projects.
> >>
> >> At the ASF, this has been a problem area called "umbrella" projects and
> over the years,
> >> all I've seen from them is wasted bandwidth, artificial barriers and
> the inventions of
> >> new ways to perform process mongering and to reduce the fun in
> developing software
> >> at this fantastic foundation.
> >>
> >> I've talked about umbrella projects enough. We've diverted conversation
> enough.
> >> Enough people have tried to act like there is some technical mumbo
> jumbo that is
> >> preventing the eventual act of higher power that I myself hope comes
> should these
> >> discussions prove unfruitful through normal means.
> >>
> >> *these. are. separate. projects.*
> >>
> *there.are.not.blocker.issues.from.spinning.out.these.projects.as.their.own.communities*
> >>
> >> In this email: http://s.apache.org/rSm
> >>
> >> And in the 2 subsequent follow ons in that thread, I've outlined a
> process that I'll copy
> >> through below for splitting these projects into their own TLPs:
> >>
> >> -----snip
> >> Process:
> >>
> >> 0. [DISCUSS] thread for <TLP name> in which you talk about #1 and #2
http://hortonworks.com/download/
+
Alejandro Abdelnur 2012-08-29, 17:13
+
Suresh Srinivas 2012-08-29, 17:26
+
Konstantin Boudnik 2012-08-29, 18:41
+
Arun C Murthy 2012-08-29, 17:04
+
Arun C Murthy 2012-08-29, 18:22
+
Eli Collins 2012-08-29, 18:49
+
Tom White 2012-08-29, 20:34
+
Alejandro Abdelnur 2012-08-29, 20:40
+
Todd Lipcon 2012-08-29, 21:18
+
Andrew Purtell 2012-08-30, 13:46
+
Sharad Agarwal 2012-08-30, 06:41
+
Mattmann, Chris A 2012-08-29, 23:29
+
Doug Cutting 2012-08-30, 16:17
+
Todd Lipcon 2012-08-29, 23:44
+
Konstantin Boudnik 2012-08-30, 02:59
+
Mattmann, Chris A 2012-08-29, 23:54
+
Konstantin Shvachko 2012-08-30, 10:12
+
Andrew Purtell 2012-08-30, 14:11
+
Aaron T. Myers 2012-08-30, 14:23
+
Brock Noland 2012-08-30, 14:43
+
Devaraj Das 2012-08-31, 01:28
+
Andrew Purtell 2012-08-31, 06:02
+
Mattmann, Chris A 2012-08-31, 06:15
+
Andrew Purtell 2012-08-31, 06:42
+
Mattmann, Chris A 2012-08-31, 06:50
+
Todd Lipcon 2012-08-31, 16:59
+
Alejandro Abdelnur 2012-08-31, 17:10
+
Jagane Sundar 2012-08-31, 17:24
+
Alejandro Abdelnur 2012-08-31, 17:11
+
Andrew Purtell 2012-08-31, 07:55
+
Robert Evans 2012-08-31, 14:34
+
Mattmann, Chris A 2012-08-31, 15:09
+
Doug Cutting 2012-08-31, 16:00
+
Mattmann, Chris A 2012-08-31, 16:08
+
Eric Baldeschwieler 2012-09-01, 00:23
+
Sharad Agarwal 2012-09-01, 09:59
+
Mahadev Konar 2012-08-31, 15:05
+
Mattmann, Chris A 2012-08-31, 06:36
+
Mattmann, Chris A 2012-08-30, 13:51
+
Arun C Murthy 2012-08-30, 10:25
+
Arun C Murthy 2012-08-30, 11:00
+
Todd Lipcon 2012-08-30, 00:16
+
Mattmann, Chris A 2012-08-30, 00:55
+
Mattmann, Chris A 2012-08-29, 23:34
+
Jakob Homan 2012-08-29, 21:22
+
Travis Thompson 2012-08-29, 22:30
+
Steve Loughran 2012-08-31, 11:54
+
Mattmann, Chris A 2012-08-29, 14:14
+
Robert Evans 2012-08-29, 15:17
+
Mattmann, Chris A 2012-08-29, 17:22
+
Eric Baldeschwieler 2012-08-29, 03:45
+
Michael Segel 2012-08-29, 17:26
+
Arun C Murthy 2012-08-29, 18:48
+
Mattmann, Chris A 2012-08-29, 23:20
+
Eli Collins 2012-08-30, 05:46
+
Arun C Murthy 2012-08-30, 06:31
+
Eli Collins 2012-08-30, 07:02
+
Alejandro Abdelnur 2012-08-30, 07:11
+
Arun C Murthy 2012-08-30, 12:29
+
Todd Lipcon 2012-08-29, 23:35
+
Aaron T. Myers 2012-08-29, 23:53
+
Konstantin Boudnik 2012-08-29, 23:47
+
Todd Lipcon 2012-08-29, 23:48
+
Arun C Murthy 2012-08-30, 01:52
+
Inder.dev Java 2012-08-30, 16:33
+
Doug Cutting 2012-08-30, 17:00
+
Owen OMalley 2012-08-30, 18:25
+
Inder.dev Java 2012-08-31, 19:00
+
Doug Cutting 2012-08-31, 20:44
+
Eric Baldeschwieler 2012-08-31, 22:43
+
Eli Collins 2012-08-29, 18:41
+
Mattmann, Chris A 2012-08-29, 23:19
+
Eli Collins 2012-08-30, 05:38
+
Mattmann, Chris A 2012-08-30, 06:06
+
Eli Collins 2012-08-30, 06:18
+
Mattmann, Chris A 2012-08-30, 06:31
+
Eli Collins 2012-08-30, 07:17
+
Konstantin Boudnik 2012-08-29, 23:27
+
Mattmann, Chris A 2012-08-29, 23:32
+
Arun C Murthy 2012-08-30, 01:47
+
Konstantin Boudnik 2012-08-29, 23:40
+
Jun Ping Du 2012-08-29, 18:35
+
Chris Douglas 2012-08-31, 01:24
+
Vinod Kumar Vavilapalli 2012-08-31, 03:35
+
Roman Shaposhnik 2012-08-31, 15:59
+
Andrew Purtell 2012-09-01, 13:21
+
Eli Collins 2012-08-31, 16:54
+
Robert Evans 2012-08-31, 16:58
+
Todd Lipcon 2012-08-31, 17:06
+
Robert Evans 2012-08-31, 18:15
+
Andrew Purtell 2012-09-01, 13:32
+
Arun C Murthy 2012-09-03, 11:02