Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce >> mail # user >> FileNotFoundException When DistributedCache file with YARN


Copy link to this message
-
FileNotFoundException When DistributedCache file with YARN
HI

 I adopt distributed cache  to implement the semi-joins.
 I am using CDH4.1.2

 the Map side setup  function is as [1]:
It works well in my eclipse indigo, howevet it goes wrong when I run it  in
cli:
The  exeception in one of the containers refers to[2].

How could I solve this exception?

Regards
[1]-------------------------------------------
@Override
public void setup(Context context){
try {
//add DistributedCache files to the Mapper.
//this DistributedCache files are on the HDFS
URI[] cacheFiles DistributedCache.getCacheFiles(context.getConfiguration());
if (cacheFiles != null && cacheFiles.length > 0){
Path path = new Path(cacheFiles[0].toString());
fs = FileSystem.get(path.toUri(),context.getConfiguration());
in = fs.open(path);
    String line ;
String[] tokens ;
    while ((line = in.readLine()) != null ) {
tokens = StringUtils.split(line,'|');
if (tokens != null && tokens.length >= 3){
joinData.put(tokens[0], tokens[1]);//key:ifid ;value:cmtsid
}
}
}
} catch (IOException e) {
e.printStackTrace();
}finally{
try {
in.close();
} catch (IOException e) {
e.printStackTrace();
}
}
}
[2]----------------------------
2013-05-13 16:11:08,444 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
Downloading public rsrc:{
hdfs://Hadoop01:8040/user/hadoop/sqoop/CMTSIFTABLE/part-m-00000,
1366969672048, FILE }
2013-05-13 16:11:08,446 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
Created localizer for container_1366976077492_0022_01_000001
2013-05-13 16:11:08,446 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
Failed to download rsrc { {
hdfs://Hadoop01:8040/user/hadoop/sqoop/CMTSIFTABLE/part-m-00000,
1366969672048, FILE
},pending,[(container_1366976077492_0022_01_000001)],9096622200855127,DOWNLOADING}
java.io.FileNotFoundException: File /tmp/nm-local-dir/filecache does not
exist
at
org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:455)
at org.apache.hadoop.fs.FileSystem.primitiveMkdir(FileSystem.java:919)
at
org.apache.hadoop.fs.DelegateToFileSystem.mkdir(DelegateToFileSystem.java:143)
at org.apache.hadoop.fs.FilterFs.mkdir(FilterFs.java:189)
at org.apache.hadoop.fs.FileContext$4.next(FileContext.java:706)
at org.apache.hadoop.fs.FileContext$4.next(FileContext.java:703)
at
org.apache.hadoop.fs.FileContext$FSLinkResolver.resolve(FileContext.java:2333)
at org.apache.hadoop.fs.FileContext.mkdir(FileContext.java:703)
at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:147)
at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:49)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB