-Unexpected Hadoop behavior: map task re-running after reducer has been running
I can't explain this behavior, can someone help me here:
Kind % Complete Num Tasks Pending Running Complete Killed Failed/Killed
map 100.00% 23547 0 1 23546 0 247 / 0
reduce 62.40% 10000 3738 30 6232 0 336 / 0
This job has been running for 48 hours and the reducer is quite a ways
through its processing. But we've hit a snag (I'm not sure what snag
The map tasks were 100% complete, none running, but now I see 1 map task
running. In a few minutes that map task will finish and I'll see "Running
map tasks" change to 0, the # of failed map tasks will increment by 1, and
the map task will run again a short time thereafter. This seems perpetual as
we had 1 map task failed at the end of the map processing when the reducer
started running, now we have 247 failed map tasks and this is the pattern
I've been watching for hours now.
Anyone want to venture some guesses here?