Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop, mail # dev - Thoughts on managing big changes to Hadoop project

Copy link to this message
Thoughts on managing big changes to Hadoop project
Eric Baldeschwieler 2012-05-08, 16:18
Hi Folks,

Owen suggested we move a discussion from HADOOP-8248 to this list.  I thought I'd repost my comment for feedback...

A community process for change management is on my mind because we are seeing a lot of major change going into things like HDFS, where the down side to user of getting something wrong can be HUGE.  Other open source communities manage these sorts of things with a more rigorous process than we have.  This is an attempt to strip down previous proposals to something that would be easy to implement if we chose to do so.


IMO the HEP (Hadoop Enhancement Process) discussion we had a while back was focused on a set of core principles I'd like us to consider again. Simple fixes and enhancements are easy, but for a major platform like Hadoop, I think we should have a community process to make sure that big changes are good changes. Before a big change goes into Hadoop, we want to know that it is complete, meaning:

• There is an understanding of the use cases / needs the change is addressing
• There is agreement that the change makes hadoop better
• The work is done: Full design / Real Tests & executed test plan / Docs
• It meets our API compatibility goals

For big changes I'd like to see us focus on defining a process that makes sure the above points are discussed and everyone agrees that the work is complete. I feel like the bylaw changes discussed here [HADOOP-8248] are focused on procedure [not outcome].

A HELP proposal (Hadoop Enhancement Lightweight Process):   ;-)

• When a proposal seems complex, folks can request that a HELP is filed...
• A doc is created in the subversion tree .../HELP/<JIRA-ID> of a companion JIRA
(often the preexisting JIRA the issue was raised on)
• Development can take place on a branch, via patches, whatever, but it is not committed until...
• The HELP document is complete (there will be a HELP template with appropriate questions)
• The help document outlines:
• the need for the change
• the use cases
• An external spec of the change (EG man page / java doc)
• An examination of API/compatibility issues
• a vote on common-dev occurs blessing the HELP DOC
• .../HELP/<JIRA-ID>.designdoc & .../HELP/<JIRA-ID>.testplan are complete
• a vote on common-dev blesses the code, the design doc, the test plan, that the work is complete...
• Beyond the votes on common-dev to guarantee wide acceptance of the change all work can happen on the JIRA

What do people think?