Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce, mail # user - Job cleanup


Copy link to this message
-
Job cleanup
Robert Dyer 2013-04-13, 19:35
What does the job cleanup task do?  My understanding was it just cleaned up
any intermediate/temporary files and moved the reducer output to the output
directory?  Does it do more?

One of my jobs runs, all maps and reduces finish, but then the job cleanup
task never finishes.  Instead it gets killed several times until the entire
Job gets killed:

Task attempt_201303272327_0772_m_000105_0 failed to report status for
600 seconds. Killing!
I suppose that since my reducers generate around 20GB of output, that
perhaps moving it takes too long?

Is it possible to disable speculative execution *only* for the cleanup task?