Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce, mail # user - Re: 答复: distcp error.


Copy link to this message
-
Re: distcp error.
Daryn Sharp 2012-08-31, 19:50
Try taking a look at your NN logs to see why it had an internal server error.  I believe the servlets are hardcoded to decode the path as UTF-8, but maybe the client used a different encoding.

Daryn

On Aug 28, 2012, at 11:30 AM, Tao wrote:

Hi,
         Thanks for your reply.
         I have tried between 1.0.3s and 2.0.1s.
         Both are failed.

         Path contain Chinese character.
         1.0.3 hftp to 1.0.3 hdfs, exception inform is below.
                                     12/08/29 00:24:23 INFO tools.DistCp: sourcePathsCount=2
12/08/29 00:24:23 INFO tools.DistCp: filesToCopyCount=1
12/08/29 00:24:23 INFO tools.DistCp: bytesToCopyCount=1.2k
12/08/29 00:24:24 INFO mapred.JobClient: Running job: job_201208101345_2203
12/08/29 00:24:25 INFO mapred.JobClient:  map 0% reduce 0%
12/08/29 00:24:46 INFO mapred.JobClient: Task Id : attempt_201208101345_2203_m_000000_0, Status : FAILED
java.io.IOException: Copied: 0 Skipped: 0 Failed: 1
        at org.apache.hadoop.tools.DistCp$CopyFilesMapper.close(DistCp.java:582)
        at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:57)
        at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:436)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372)
        at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:396)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
        at org.apache.hadoop.mapred.Child.main(Child.java:249)

12/08/29 00:25:04 INFO mapred.JobClient: Task Id : attempt_201208101345_2203_m_000000_1, Status : FAILED
java.io.IOException: Copied: 0 Skipped: 0 Failed: 1
        at org.apache.hadoop.tools.DistCp$CopyFilesMapper.close(DistCp.java:582)
        at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:57)
        at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:436)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372)
        at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:396)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
        at org.apache.hadoop.mapred.Child.main(Child.java:249)

12/08/29 00:25:19 INFO mapred.JobClient: Task Id : attempt_201208101345_2203_m_000000_2, Status : FAILED
java.io.IOException: Copied: 0 Skipped: 0 Failed: 1
        at org.apache.hadoop.tools.DistCp$CopyFilesMapper.close(DistCp.java:582)
        at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:57)
        at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:436)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372)
        at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:396)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
        at org.apache.hadoop.mapred.Child.main(Child.java:249)

12/08/29 00:25:40 INFO mapred.JobClient: Job complete: job_201208101345_2203
12/08/29 00:25:40 INFO mapred.JobClient: Counters: 6
12/08/29 00:25:40 INFO mapred.JobClient:   Job Counters
12/08/29 00:25:40 INFO mapred.JobClient:     SLOTS_MILLIS_MAPS=66844
12/08/29 00:25:40 INFO mapred.JobClient:     Total time spent by all reduces waiting after reserving slots (ms)=0
12/08/29 00:25:40 INFO mapred.JobClient:     Total time spent by all maps waiting after reserving slots (ms)=0
12/08/29 00:25:40 INFO mapred.JobClient:     Launched map tasks=4
12/08/29 00:25:40 INFO mapred.JobClient:     SLOTS_MILLIS_REDUCES=0
12/08/29 00:25:40 INFO mapred.JobClient:     Failed map tasks=1
12/08/29 00:25:40 INFO mapred.JobClient: Job Failed: # of failed Map Tasks exceeded allowed limit. FailedCount: 1. LastFailedTask: task_201208101345_2203_m_000000
With failures, global counters are inaccurate; consider running with -i
Copy failed: java.io.IOException: Job failed!
        at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1265)
        at org.apache.hadoop.tools.DistCp.copy(DistCp.java:667)
        at org.apache.hadoop.tools.DistCp.run(DistCp.java:881)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
        at org.apache.hadoop.tools.DistCp.main(DistCp.java:908)
         2.0.1 hftp to 2.0.1 hdfs, exception inform is below.
12/08/29 00:20:06 INFO tools.DistCp: DistCp job-id: job_1345831938927_0043
12/08/29 00:20:06 INFO mapreduce.Job: Running job: job_1345831938927_0043
12/08/29 00:20:14 INFO mapreduce.Job: Job job_1345831938927_0043 running in uber mode : false
12/08/29 00:20:14 INFO mapreduce.Job:  map 0% reduce 0%
12/08/29 00:20:23 INFO mapreduce.Job: Task Id : attempt_1345831938927_0043_m_000000_0, Status : FAILED
Error: java.io.IOException: File copy failed: hftp://baby20:50070/tmp/??.log/add.csv --> hdfs://baby20:54310/tmp4/add.csv
        at org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:262)
        at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:229)
        at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:45)
        at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
        at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:725)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332)
        at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:152)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:396)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)
        at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:147)
Caused by: