Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # user >> Unexpected Hadoop behavior: map task re-running after reducer has been running


Copy link to this message
-
Unexpected Hadoop behavior: map task re-running after reducer has been running
I can't explain this behavior, can someone help me here:

 

  Kind  % Complete Num Tasks Pending Running Complete Killed Failed/Killed
Task Attempts

    map      100.00%    23547       0       1    23546      0       247 / 0

   reduce     62.40%    10000    3738      30     6232      0       336 / 0

 

This job has been running for 48 hours and the reducer is quite a ways
through its processing. But we've hit a snag (I'm not sure what snag
exactly).

 

The map tasks were 100% complete, none running, but now I see 1 map task
running. In a few minutes that map task will finish and I'll see "Running
map tasks" change to 0, the # of failed map tasks will increment by 1, and
the map task will run again a short time thereafter. This seems perpetual as
we had 1 map task failed at the end of the map processing when the reducer
started running, now we have 247 failed map tasks and this is the pattern
I've been watching for hours now.

 

Anyone want to venture some guesses here?

 

Thanks,

David

 

 

 

 

 

NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB