Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HDFS >> mail # user >> Exception during reduce phase when running jobs remotely


Copy link to this message
-
Exception during reduce phase when running jobs remotely
I have a small hadoop cluster running 1.0.4 and I'm trying to have it setup
so that I can run jobs remotely from a computer on the same network that is
not a part of the cluster. I've got a main java class that
implements org.apache.hadoop.util.Tool and I'm able to run this job from
the NameNode using ToolRunner.run(), setting up the JobConf, and submitting
with JobClient.submitJob().

When I try to run the same class remotely from any machine that is not the
NameNode the job is submitted and it appears that the Map tasks
successfully complete, but I get the following exception for all of the
reduce tasks:

org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find
output/map_0.out in any of the configured local directories
at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathToRead(LocalDirAllocator.java:429)
at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathToRead(LocalDirAllocator.java:160)
at org.apache.hadoop.mapred.MapOutputFile.getInputFile(MapOutputFile.java:161)
at org.apache.hadoop.mapred.ReduceTask.getMapFiles(ReduceTask.java:220)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:398)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
I'm not sure how to interpret that error message. I think I'm missing
some config files that are not present on the remote machine but I
can't pin down exactly what I need. Does anyone have any guidance on
what the above error means or how to submit jobs remotely?

Thanks,

--Oren
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB