Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
MapReduce, mail # user - Re: Multi-stage map/reduce jobs


+
Bertrand Dechoux 2012-11-24, 11:56
Copy link to this message
-
Multi-stage map/reduce jobs
Sean McNamara 2012-11-23, 22:22
It's not clear to me how to stitch together multiple map reduce jobs.  Without using cascading or something else like it, is the method basically to write to a intermediate spot, and have the next stage read from there?

If so, how are jobs responsible for cleaning up the temp/intermediate data they create?  What happens if stage 1 completes, and state 2 doesn't, do the stage 1 files get left around?

Does anyone have some insight they could share?

Thanks.
+
Jay Vyas 2012-11-23, 22:50