Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop, mail # general - [DISCUSSION] Combine committer lists in Common/HDFS/MapReduce


Copy link to this message
-
Re: [DISCUSSION] Combine committer lists in Common/HDFS/MapReduce
Chris Douglas 2010-08-14, 02:41
On Tue, Aug 10, 2010 at 10:35 PM, Vinod KV <[EMAIL PROTECTED]> wrote:
>
> I know of very few to nearly zero number of patches in mapreduce that touch
> HDFS also. And vice versa. The common case is of patches that touch both
> common and mapreduce or common and hdfs.

This is a good point, though changes in Common and HDFS interfaces do
break MapReduce, particularly unit tests that take liberties with
interface visibility. It would be convenient if HDFS committers could
push in the fix with the original issue, to shrink the window where MR
is broken, but in practice such changes are usually committed in short
order. After all, most committers have commit access to all three
projects... though this is one of the reasons why the constraint
difficult to justify.

> I am one of those who has access to
> mapreduce project but not to common project and find more than 50% of the
> patches that fall into this category.

This *was* an oversight that should be corrected either through this
vote or independently.

> As for the separation of the repositories, I personally felt separation of
> mapreduce from hdfs helped focusing on things a lot better. The last gasp
> work done for 0.21, mostly by Tom, did help a lot in decoupling the
> projects. Common is the hot point, sure, but as others noted, that is a
> separate discussion.

+1

----

The discussion appears to be dying down. Quick summary of comments so far:

* That all HDFS and MapReduce committers should have commit rights to
Common appears to be undisputed.

* The value of splitting the projects at all is disputed. Some have
argued that it has complicated work without delivering the benefits to
developers it promised, though others have experienced this as
discipline rather than inconvenience. Most of these complaints
reference patches that touch Common, particularly actively-developed
packages like fs. The vote concerns a narrower question than the
project split, though it's fair to assert a lack of unanimity on the
premise of a split Hadoop, let alone the more limited question of
whether to retain a split list of committers.

* Not everyone agrees that combining HDFS and MapReduce committers is
sound. While there is sufficient overlap today to branch all three
together, patch releases could- and likely will- be cut independently.
Not everyone thinks the two projects are developing independent
communities, but none have difficulty imagining TLP status for both.

Please feel free to amend this if I left out an important point.

-----

I think we're ready to vote. Though we have no bylaws to amend, this
would be a modification to them, I guess. The last-proposed set
required a 2/3 majority of the PMC, IIRC. Since adding a committer
requires consensus on the PMC, it's probably fair to require a 2/3
majority to cross-pollenate lists instead of a simple majority.

Though the vote could be conducted on a huge cross-post to
mapreduce-dev@, hdfs-dev@ and common-dev@, it'll be easier to count if
it's run on general@; I'll start it here on Monday if nobody minds. -C