Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # user >> RuntimeException, coult not obtain block causes trackers to get blacklisted


Copy link to this message
-
RuntimeException, coult not obtain block causes trackers to get blacklisted
Hello,
I have an intensive job running across 5 machines. During the map
stage, each map emits 200 records, so effectively for 50,000,000 input
reords, the map creates 200*50e6 records.

However, after a long time, I see two trackers are blacklisted
Caused by: java.lang.RuntimeException: Could not obtain block:
blk_-5964245027287878843_92134
file=/tmp/8a5bc814-b4ff-4641-bc3a-abfeda9e7e33.mapfile/index
        at org.saptarshiguha.rhipe.hadoop.RHMR$RHMRCombiner.configure(RHMR.java:314)
        at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:58)
        at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:83)
        at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.combineAndSpill(MapTask.java:1110)
        at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:989)
        at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.access$1800(MapTask.java:401)
        at org.apache.hadoop.mapred.MapTask$MapOutputBuffer$SpillThread.run(MapTask.java:886)
The map file is very much present in that location and other trackers
could read it. What could be the reason for these two machines not
being able to read it?
Now that they are blacklisted, how can I add them back to the computation (a
Saptarshi Guha
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB