Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
HDFS, mail # user - FileNotFoundExcepion when getting files from DistributedCache


Copy link to this message
-
FileNotFoundExcepion when getting files from DistributedCache
Barak Yaish 2012-11-22, 20:34
Hi,

I’ve 2 nodes cluster (v1.04), master and slave. On the master, in
Tool.run() we add two files to the DistributedCache using addCacheFile().
Files do exist in HDFS. In the Mapper.setup() we want to retrieve those
files from the cache using FSDataInputStream fs = FileSystem.get(
context.getConfiguration() ).open( path ). The problem is that for one file
a FileNotFoundException is thrown, although the file exists on the slave
node:

attempt_201211211227_0020_m_000000_2: java.io.FileNotFoundException: File
does not exist:
/somedir/hdp.tmp.dir/mapred/local/taskTracker/distcache/-7769715304990780/master/tmp/analytics/1.csv

ls –l on the slave:

[hduser@slave ~]$ ll
/somedir/hdp.tmp.dir/mapred/local/taskTracker/distcache/-7769715304990780/master/tmp/
analytics/1.csv
-rwxr-xr-x 1 hduser hadoop 42701 Nov 22 10:18
/somedir/hdp.tmp.dir/mapred/local/taskTracker/distcache/-7769715304990780/master/tmp/analytics/1.csv
[hduser@slave ~]$

My questions are:

   1. Shouldn't all files exist on all nodes?
   2. What should be done to fix that?

Thanks.
+
Harsh J 2012-11-22, 20:38
+
Barak Yaish 2012-11-22, 20:46
+
Harsh J 2012-11-22, 20:50
+
Barak Yaish 2012-11-22, 21:09