Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # user >> RE: distcp failed "Copy failed: ENOENT: No such file or directory"

Copy link to this message
RE: distcp failed "Copy failed: ENOENT: No such file or directory"
The error doesn't mean the file not existed in the HDFS, but it means local disk. If you read the error stack trace:
at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:581)
It indicates the error happened on Local file system.
If you try to copy data from an existing cluster to a new setup cluster, then It most likely some misconfiguration in your new cluster causing this.

When you use the distcp command, hadoop will generate a lot of MR jobs to copy the data in Mapper tasks. (No reducers in this case). You should check the JT to find out which task node gave this error, and check the log to identify the root cause, especially check the local.dir or log.dir settings in the new cluster.

Date: Fri, 6 Sep 2013 15:36:48 +0800
Subject: distcp failed "Copy failed: ENOENT: No such file or directory"

# sudo -u hdfs hadoop distcp hdfs:// hdfs://
13/09/06 15:34:11 INFO tools.DistCp: srcPaths=[hdfs://]

13/09/06 15:34:11 INFO tools.DistCp: destPath=hdfs://
13/09/06 15:34:12 WARN conf.Configuration: session.id is deprecated. Instead, use dfs.metrics.session-id

13/09/06 15:34:12 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId13/09/06 15:34:12 WARN conf.Configuration: slave.host.name is deprecated. Instead, use dfs.datanode.hostname

With failures, global counters are inaccurate; consider running with -i
Copy failed: ENOENT: No such file or directory
        at org.apache.hadoop.io.nativeio.NativeIO.chmod(Native Method)
        at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:581)

        at org.apache.hadoop.fs.FilterFileSystem.setPermission(FilterFileSystem.java:427)
        at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:579)
        at org.apache.hadoop.tools.DistCp.setup(DistCp.java:1038)

        at org.apache.hadoop.tools.DistCp.copy(DistCp.java:666)
        at org.apache.hadoop.tools.DistCp.run(DistCp.java:881)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)

        at org.apache.hadoop.tools.DistCp.main(DistCp.java:908)
the src and dest dir is all exist
# sudo -u hdfs hadoop fs -ls /alex
Found 3 items
drwxr-xr-x   - root supergroup          0 2013-08-16 10:14 /alex/mydump
drwxr-xr-x   - root supergroup          0 2013-08-28 10:48 /alex/tdump
drwxr-xr-x   - root supergroup          0 2013-08-28 10:54 /alex/tdump2
#  sudo -u hdfs hadoop fs -ls /benchmarks
Found 1 items
drwxr-xr-x   - hdfs hdfs          0 2013-08-23 17:02 /benchmarks/TestDFSIO