Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce >> mail # user >> Re: 答复: distcp error.


Copy link to this message
-
Re: distcp error.
Try taking a look at your NN logs to see why it had an internal server error.  I believe the servlets are hardcoded to decode the path as UTF-8, but maybe the client used a different encoding.

Daryn

On Aug 28, 2012, at 11:30 AM, Tao wrote:

Hi,
         Thanks for your reply.
         I have tried between 1.0.3s and 2.0.1s.
         Both are failed.

         Path contain Chinese character.
         1.0.3 hftp to 1.0.3 hdfs, exception inform is below.
                                     12/08/29 00:24:23 INFO tools.DistCp: sourcePathsCount=2
12/08/29 00:24:23 INFO tools.DistCp: filesToCopyCount=1
12/08/29 00:24:23 INFO tools.DistCp: bytesToCopyCount=1.2k
12/08/29 00:24:24 INFO mapred.JobClient: Running job: job_201208101345_2203
12/08/29 00:24:25 INFO mapred.JobClient:  map 0% reduce 0%
12/08/29 00:24:46 INFO mapred.JobClient: Task Id : attempt_201208101345_2203_m_000000_0, Status : FAILED
java.io.IOException: Copied: 0 Skipped: 0 Failed: 1
        at org.apache.hadoop.tools.DistCp$CopyFilesMapper.close(DistCp.java:582)
        at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:57)
        at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:436)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372)
        at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:396)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
        at org.apache.hadoop.mapred.Child.main(Child.java:249)

12/08/29 00:25:04 INFO mapred.JobClient: Task Id : attempt_201208101345_2203_m_000000_1, Status : FAILED
java.io.IOException: Copied: 0 Skipped: 0 Failed: 1
        at org.apache.hadoop.tools.DistCp$CopyFilesMapper.close(DistCp.java:582)
        at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:57)
        at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:436)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372)
        at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:396)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
        at org.apache.hadoop.mapred.Child.main(Child.java:249)

12/08/29 00:25:19 INFO mapred.JobClient: Task Id : attempt_201208101345_2203_m_000000_2, Status : FAILED
java.io.IOException: Copied: 0 Skipped: 0 Failed: 1
        at org.apache.hadoop.tools.DistCp$CopyFilesMapper.close(DistCp.java:582)
        at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:57)
        at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:436)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372)
        at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:396)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
        at org.apache.hadoop.mapred.Child.main(Child.java:249)

12/08/29 00:25:40 INFO mapred.JobClient: Job complete: job_201208101345_2203
12/08/29 00:25:40 INFO mapred.JobClient: Counters: 6
12/08/29 00:25:40 INFO mapred.JobClient:   Job Counters
12/08/29 00:25:40 INFO mapred.JobClient:     SLOTS_MILLIS_MAPS=66844
12/08/29 00:25:40 INFO mapred.JobClient:     Total time spent by all reduces waiting after reserving slots (ms)=0
12/08/29 00:25:40 INFO mapred.JobClient:     Total time spent by all maps waiting after reserving slots (ms)=0
12/08/29 00:25:40 INFO mapred.JobClient:     Launched map tasks=4
12/08/29 00:25:40 INFO mapred.JobClient:     SLOTS_MILLIS_REDUCES=0
12/08/29 00:25:40 INFO mapred.JobClient:     Failed map tasks=1
12/08/29 00:25:40 INFO mapred.JobClient: Job Failed: # of failed Map Tasks exceeded allowed limit. FailedCount: 1. LastFailedTask: task_201208101345_2203_m_000000
With failures, global counters are inaccurate; consider running with -i
Copy failed: java.io.IOException: Job failed!
        at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1265)
        at org.apache.hadoop.tools.DistCp.copy(DistCp.java:667)
        at org.apache.hadoop.tools.DistCp.run(DistCp.java:881)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
        at org.apache.hadoop.tools.DistCp.main(DistCp.java:908)
         2.0.1 hftp to 2.0.1 hdfs, exception inform is below.
12/08/29 00:20:06 INFO tools.DistCp: DistCp job-id: job_1345831938927_0043
12/08/29 00:20:06 INFO mapreduce.Job: Running job: job_1345831938927_0043
12/08/29 00:20:14 INFO mapreduce.Job: Job job_1345831938927_0043 running in uber mode : false
12/08/29 00:20:14 INFO mapreduce.Job:  map 0% reduce 0%
12/08/29 00:20:23 INFO mapreduce.Job: Task Id : attempt_1345831938927_0043_m_000000_0, Status : FAILED
Error: java.io.IOException: File copy failed: hftp://baby20:50070/tmp/??.log/add.csv --> hdfs://baby20:54310/tmp4/add.csv
        at org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:262)
        at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:229)
        at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:45)
        at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
        at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:725)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332)
        at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:152)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:396)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)
        at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:147)
Caused by:
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB