Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
MapReduce, mail # dev - Status on MAPREDUCE-279 and plans forward


+
Arun C Murthy 2011-06-10, 19:44
Copy link to this message
-
Re: Status on MAPREDUCE-279 and plans forward
Tom White 2011-06-11, 00:58
Hi Arun,

Great to hear that MR2 is nearly ready for trunk.

MR2 is being built using Maven, so step 5 is about getting Maven to
build all of MR I presume? The svn unsplit (HADOOP-7106) and
Mavenization work in HADOOP-6671 should make this approach more
tractable. Or were you thinking that there would be a hybrid Ant/Maven
build during a transition period?

Also, for step 3, what did you think of the directory layout I
suggested in http://bit.ly/mr-279-maven-layout?

Cheers,
Tom

On Fri, Jun 10, 2011 at 12:44 PM, Arun C Murthy <[EMAIL PROTECTED]> wrote:
> Another heads up.
>
> Things are going well, we have done scale testing (350 nodes) and continue
> to do more.
>
> We think we are nearly ready to merge MR-279 into trunk (yaay!).
>
> A proposal for plan forward:
>
> 1. I'd like to get MAPREDUCE-2400 committed to both trunk and MR-279. This
> is a very important patch by Tom which allows to submit jobs to both JT and
> the new RM - thanks Tom!
> 2. Check-in the 'yarn' subfolder which is just the ResourceManager and
> NodeManager i.e. compute fabric to trunk.
> 3. Rework the src tree in trunk to look like the MR-279 branch to ensure no
> changes to trunk are lost - this will ensure separation between the
> 'classic' framework (JT/TT) and the MR runtime (map/reduce/sort/shuffle
> etc.)
> 4. Re-play the 4-5 patches needed to the MR runtime from the MR-279 branch.
> 5. Change build scripts to reflect the new reality.
>
> In terms of timelines, I'm hoping we can get this done in the coming 2-3
> weeks.
>
> Thoughts?
>
> thanks,
> Arun
>
+
Arun C Murthy 2011-06-11, 08:10