Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Hadoop >> mail # general >> [DISCUSS] Spin out MR, HDFS and YARN as their own TLPs and disband Hadoop umbrella project


+
Mattmann, Chris A 2012-08-29, 02:33
+
Alejandro Abdelnur 2012-08-29, 03:50
+
Arun C Murthy 2012-08-29, 16:31
+
Tom White 2012-08-29, 17:30
+
Eric Baldeschwieler 2012-08-29, 17:42
+
Suresh Srinivas 2012-08-29, 17:02
+
Alejandro Abdelnur 2012-08-29, 17:13
+
Suresh Srinivas 2012-08-29, 17:26
+
Konstantin Boudnik 2012-08-29, 18:41
+
Arun C Murthy 2012-08-29, 17:04
+
Arun C Murthy 2012-08-29, 18:22
+
Eli Collins 2012-08-29, 18:49
+
Tom White 2012-08-29, 20:34
+
Alejandro Abdelnur 2012-08-29, 20:40
+
Todd Lipcon 2012-08-29, 21:18
+
Andrew Purtell 2012-08-30, 13:46
+
Sharad Agarwal 2012-08-30, 06:41
+
Mattmann, Chris A 2012-08-29, 23:29
+
Doug Cutting 2012-08-30, 16:17
Copy link to this message
-
Re: [DISCUSS] Spin out MR, HDFS and YARN as their own TLPs and disband Hadoop umbrella project
On Wed, Aug 29, 2012 at 4:29 PM, Mattmann, Chris A (388J)
<[EMAIL PROTECTED]> wrote:

> You're right, it's not project boundaries, it's poor community behavior,
> and general umbrella-project-ness.

No doubt there's bad behavior. But splitting into smaller projects
won't help anything. We'll still have the exact same behavior inside
the smaller projects.

>
> One aspect I've seen is that exclusivity of allowing people to become
> PMC members on the project, and the separation of PMC from C.
> Other things I've seen are the use of technical justifications or complexity
> issues as an excuse for the exclusivity, as an excuse for drawing boundaries
> between project committers and PMC members, and then between specific
> products that the project and community as a whole releases, and finally
> other things I've seen include external interests influencing the way that
> business is done around here (need for releases in downstream companies,
> or projects driving upstream, Apache decisions, which are supposed to be
> independent of any lone company, or set of companies -- it's individuals here
> that do the work).
>

It's individuals that do the work, but the individuals get paid by
companies, so individuals acting in their best interests are going to
tend to align with their company. They also often know details about
their customer bases that they can't share directly, which can be
frustrating, but it's a fact of life. I'm sure we'd see the same if we
were 20 independent consultants each with our own priorities, etc.

> The above is not a discrete thing that's happened once, or twice, or that
> happened three times, but was fixed later. It's never been fixed.
>

IMO it's massively improved since a couple years ago. We're making
good progress on the 2.0 line, we no longer have divergent forks, and
I haven't seen an issue get vetoed in recent memory. Please provide
some recent examples where you think that splitting into smaller
granularity projects would help anything.

>>
>> Instead, the issues are usually _within_ a component. So, if we split
>> into 3 TLPs, then we'll just have 3 TLPs, each of which is just as
>> contentious as before.
>
> I doubt that. Creating TLPs either directly by going to the board, or
> via going to the Incubator should involve a set of members of the
> committee (PMC) that desire to work together; that ideally trust one another; that
> are inclusive to others who jump on the list and discuss things; and that
> collect these principles into the "Apache way", and build and deliver software at
> no cost to the public via this Foundation.

Just because we argue doesn't mean we don't desire to work together.
Smart passionate people will argue. I argue with my colleagues here at
Cloudera, I argue with Hortonworkers, and I argue with Facebookers -
it doesn't really matter much. I still enjoy getting beers with them
when I end up at conferences. No hard feelings, we're all adults,
right?

>
> Currently, the Apache Hadoop project isn't doing that. Something needs
> to be done to fix it. Just because an attempt to split the projects in the past
> didn't work doesn't mean that the Hadoop community should just accept
> "this is a popular project; it's going to be contentious; nothing to see here
> folks".

Again, please provide examples. From my vantage point, I see a lot of
progress being made on critical features: we've done federation, HA
namenode, massive performance improvements, YARN, practically
rewritten NameNode, and more in the last couple years. Hardly an
unproductive community.

>
> It's more than that.
>
>>
>> Let's just embrace contention as a fact of life on a high-profile
>> high-stakes project and get back to work.
>
> -1 to that. Apache projects shouldn't be contentious, whether you are a billion dollar
> industry like Hadoop, or whether you are the US govt, or whether you are Joe Blow,
> Mom and Pop, building software to deliver to food truck vendors. It doesn't matter.
> Period.

I guess we'll have to agree to disagree.
...says the guy who isn't on the hook to stitch it all back together
into a deliverable for demanding customers, maintain green Jenkins
builds, etc. You can say these aren't technical issues, but if you're
not dealing with the project on a technical basis, I don't think
you're well qualified to judge. I certainly appreciate the work you've
done way back in the Nutch days and your continued evangelism, but
this whole thread just seems like it's stirring up trouble and not
going to accomplish anything except a bunch of wasted man-hours. (I've
already wasted about 45 minutes today on it, oops!)

-Todd
Todd Lipcon
Software Engineer, Cloudera
+
Konstantin Boudnik 2012-08-30, 02:59
+
Mattmann, Chris A 2012-08-29, 23:54
+
Konstantin Shvachko 2012-08-30, 10:12
+
Andrew Purtell 2012-08-30, 14:11
+
Aaron T. Myers 2012-08-30, 14:23
+
Brock Noland 2012-08-30, 14:43
+
Devaraj Das 2012-08-31, 01:28
+
Andrew Purtell 2012-08-31, 06:02
+
Mattmann, Chris A 2012-08-31, 06:15
+
Andrew Purtell 2012-08-31, 06:42
+
Mattmann, Chris A 2012-08-31, 06:50
+
Todd Lipcon 2012-08-31, 16:59
+
Alejandro Abdelnur 2012-08-31, 17:10
+
Jagane Sundar 2012-08-31, 17:24
+
Alejandro Abdelnur 2012-08-31, 17:11
+
Andrew Purtell 2012-08-31, 07:55
+
Robert Evans 2012-08-31, 14:34
+
Mattmann, Chris A 2012-08-31, 15:09
+
Doug Cutting 2012-08-31, 16:00
+
Mattmann, Chris A 2012-08-31, 16:08
+
Eric Baldeschwieler 2012-09-01, 00:23
+
Sharad Agarwal 2012-09-01, 09:59
+
Mahadev Konar 2012-08-31, 15:05
+
Mattmann, Chris A 2012-08-31, 06:36
+
Mattmann, Chris A 2012-08-30, 13:51
+
Arun C Murthy 2012-08-30, 10:25
+
Arun C Murthy 2012-08-30, 11:00
+
Todd Lipcon 2012-08-30, 00:16
+
Mattmann, Chris A 2012-08-30, 00:55
+
Mattmann, Chris A 2012-08-29, 23:34
+
Jakob Homan 2012-08-29, 21:22
+
Travis Thompson 2012-08-29, 22:30
+
Steve Loughran 2012-08-31, 11:54
+
Mattmann, Chris A 2012-08-29, 14:14
+
Robert Evans 2012-08-29, 15:17
+
Mattmann, Chris A 2012-08-29, 17:22
+
Eric Baldeschwieler 2012-08-29, 03:45
+
Michael Segel 2012-08-29, 17:26
+
Arun C Murthy 2012-08-29, 18:48
+
Mattmann, Chris A 2012-08-29, 23:20
+
Eli Collins 2012-08-30, 05:46
+
Arun C Murthy 2012-08-30, 06:31
+
Eli Collins 2012-08-30, 07:02
+
Alejandro Abdelnur 2012-08-30, 07:11
+
Arun C Murthy 2012-08-30, 12:29
+
Todd Lipcon 2012-08-29, 23:35
+
Aaron T. Myers 2012-08-29, 23:53
+
Konstantin Boudnik 2012-08-29, 23:47
+
Todd Lipcon 2012-08-29, 23:48
+
Arun C Murthy 2012-08-30, 01:52
+
Inder.dev Java 2012-08-30, 16:33
+
Doug Cutting 2012-08-30, 17:00
+
Owen OMalley 2012-08-30, 18:25
+
Inder.dev Java 2012-08-31, 19:00
+
Doug Cutting 2012-08-31, 20:44
+
Eric Baldeschwieler 2012-08-31, 22:43
+
Eli Collins 2012-08-29, 18:41
+
Mattmann, Chris A 2012-08-29, 23:19
+
Eli Collins 2012-08-30, 05:38
+
Mattmann, Chris A 2012-08-30, 06:06
+
Eli Collins 2012-08-30, 06:18
+
Mattmann, Chris A 2012-08-30, 06:31
+
Eli Collins 2012-08-30, 07:17
+
Konstantin Boudnik 2012-08-29, 23:27
+
Mattmann, Chris A 2012-08-29, 23:32
+
Arun C Murthy 2012-08-30, 01:47
+
Konstantin Boudnik 2012-08-29, 23:40
+
Jun Ping Du 2012-08-29, 18:35
+
Chris Douglas 2012-08-31, 01:24
+
Vinod Kumar Vavilapalli 2012-08-31, 03:35
+
Roman Shaposhnik 2012-08-31, 15:59
+
Andrew Purtell 2012-09-01, 13:21
+
Eli Collins 2012-08-31, 16:54
+
Robert Evans 2012-08-31, 16:58
+
Todd Lipcon 2012-08-31, 17:06
+
Robert Evans 2012-08-31, 18:15
+
Andrew Purtell 2012-09-01, 13:32
+
Arun C Murthy 2012-09-03, 11:02