Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop, mail # general - Proposal: Further Project Split(s)


Copy link to this message
-
Proposal: Further Project Split(s)
Aaron T. Myers 2011-04-01, 08:19
Hello Hadoop Community,

Given the tremendous positive feedback we've all had regarding the HDFS,
MapReduce, and Common project split, I'd like to propose we take the next
step and further separate the existing projects.

I propose we begin by splitting the MapReduce project into separate "Map"
and "Reduce" sub-projects. This will provide us the opportunity to tease out
the complex interdependencies between "map" and "reduce" that exist today,
to encourage us to write more modular and isolated code, which should speed
releases. This will also aid our users who exclusively run map-only or
reduce-only jobs. These are important use-cases, and so should be given high
priority.

Given that these two portions of the existing MapReduce project share a
great deal of code, we will likely need to release these two new projects
concurrently at first, but the eventual goal should certainly be to be able
to release "Map" and "Reduce" independently. This seems intuitive to me,
given the remarkable recent advancements in the academic community regarding
"reduce," while the research coming out of the "map" academics has largely
stagnated of late.

If this proposal is accepted, and it has the success I think it will, then
we should strongly consider splitting the other two projects as well. My gut
instinct is that we should split "HDFS" into "HD" and "FS" sub-projects, and
simply rename the "Common" project to "C'Mon." We can think about the
details of what exactly these project splits mean later.

Please let me know what you think.

Best,
Aaron