+
Tao 2012-08-28, 15:36
-
Re: distcp error.Marcos Ortiz 2012-08-28, 15:52
Hi, Tao. This problem is only with 2.0.1 or with the two versions?
Have you tried to use distcp from 1.0.3 to 1.0.3?
El 28/08/2012 11:36, Tao escribi锟斤拷:
>
> Hi, all
>
> I use distcp copying data from hadoop1.0.3 to hadoop 2.0.1.
>
> When the file path(or file name) contain Chinese character, an
> exception will throw. Like below. I need some help about this.
>
> Thanks.
>
> [hdfs@host ~]$ hadoop distcp -i -prbugp -m 14 -overwrite -log
> /tmp/distcp.log hftp://10.xx.xx.aa:50070/tmp/锟斤拷锟斤拷路锟斤拷锟斤拷锟斤拷hdfs:
> //10.xx.xx.bb:54310/tmp/distcp_test14
>
> 12/08/28 23:32:31 INFO tools.DistCp: Input Options:
> DistCpOptions{atomicCommit=false, syncFolder=false,
> deleteMissing=false, ignoreFailures=true, maxMaps=14,
> sslConfigurationFile='null', copyStrategy='uniformsize',
> sourceFileListing=null, sourcePaths=[hftp://10.xx.xx.aa:50070/tmp/锟斤拷锟斤拷
> 路锟斤拷锟斤拷锟斤拷], targetPath=hdfs://10.xx.xx.bb:54310/tmp/distcp_test14}
>
> 12/08/28 23:32:33 INFO tools.DistCp: DistCp job log path: /tmp/distcp.log
>
> 12/08/28 23:32:34 WARN conf.Configuration: io.sort.mb is deprecated.
> Instead, use mapreduce.task.io.sort.mb
>
> 12/08/28 23:32:34 WARN conf.Configuration: io.sort.factor is
> deprecated. Instead, use mapreduce.task.io.sort.factor
>
> 12/08/28 23:32:34 WARN util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes
> where applicable
>
> 12/08/28 23:32:36 INFO mapreduce.JobSubmitter: number of splits:1
>
> 12/08/28 23:32:36 WARN conf.Configuration: mapred.jar is deprecated.
> Instead, use mapreduce.job.jar
>
> 12/08/28 23:32:36 WARN conf.Configuration:
> mapred.map.tasks.speculative.execution is deprecated. Instead, use
> mapreduce.map.speculative
>
> 12/08/28 23:32:36 WARN conf.Configuration: mapred.reduce.tasks is
> deprecated. Instead, use mapreduce.job.reduces
>
> 12/08/28 23:32:36 WARN conf.Configuration:
> mapred.mapoutput.value.class is deprecated. Instead, use
> mapreduce.map.output.value.class
>
> 12/08/28 23:32:36 WARN conf.Configuration: mapreduce.map.class is
> deprecated. Instead, use mapreduce.job.map.class
>
> 12/08/28 23:32:36 WARN conf.Configuration: mapred.job.name is
> deprecated. Instead, use mapreduce.job.name
>
> 12/08/28 23:32:36 WARN conf.Configuration: mapreduce.inputformat.class
> is deprecated. Instead, use mapreduce.job.inputformat.class
>
> 12/08/28 23:32:36 WARN conf.Configuration: mapred.output.dir is
> deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir
>
> 12/08/28 23:32:36 WARN conf.Configuration:
> mapreduce.outputformat.class is deprecated. Instead, use
> mapreduce.job.outputformat.class
>
> 12/08/28 23:32:36 WARN conf.Configuration: mapred.map.tasks is
> deprecated. Instead, use mapreduce.job.maps
>
> 12/08/28 23:32:36 WARN conf.Configuration: mapred.mapoutput.key.class
> is deprecated. Instead, use mapreduce.map.output.key.class
>
> 12/08/28 23:32:36 WARN conf.Configuration: mapred.working.dir is
> deprecated. Instead, use mapreduce.job.working.dir
>
> 12/08/28 23:32:37 INFO mapred.ResourceMgrDelegate: Submitted
> application application_1345831938927_0039 to ResourceManager at
> baby20/10.1.1.40:8040
>
> 12/08/28 23:32:37 INFO mapreduce.Job: The url to track the job:
> http://baby20:8088/proxy/application_1345831938927_0039/
>
> 12/08/28 23:32:37 INFO tools.DistCp: DistCp job-id: job_1345831938927_0039
>
> 12/08/28 23:32:37 INFO mapreduce.Job: Running job: job_1345831938927_0039
>
> 12/08/28 23:32:50 INFO mapreduce.Job: Job job_1345831938927_0039
> running in uber mode : false
>
> 12/08/28 23:32:50 INFO mapreduce.Job: map 0% reduce 0%
>
> 12/08/28 23:33:00 INFO mapreduce.Job: map 100% reduce 0%
>
> 12/08/28 23:33:00 INFO mapreduce.Job: Task Id :
> attempt_1345831938927_0039_m_000000_0, Status : FAILED
>
> Error: java.io.IOException: File copy failed: hftp://10.1.1.26:50070
> /tmp/锟斤拷锟斤拷路锟斤拷锟斤拷锟斤拷/part-r-00017 -->
> hdfs://10.1.1.40:54310/tmp/distcp_test14/part-r-00017
>
> at
> org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:262)
10mo. ANIVERSARIO DE LA CREACION DE LA UNIVERSIDAD DE LAS CIENCIAS INFORMATICAS...
CONECTADOS AL FUTURO, CONECTADOS A LA REVOLUCION
http://www.uci.cu
http://www.facebook.com/universidad.uci
http://www.flickr.com/photos/universidad_uci
Have you tried to use distcp from 1.0.3 to 1.0.3?
El 28/08/2012 11:36, Tao escribi锟斤拷:
>
> Hi, all
>
> I use distcp copying data from hadoop1.0.3 to hadoop 2.0.1.
>
> When the file path(or file name) contain Chinese character, an
> exception will throw. Like below. I need some help about this.
>
> Thanks.
>
> [hdfs@host ~]$ hadoop distcp -i -prbugp -m 14 -overwrite -log
> /tmp/distcp.log hftp://10.xx.xx.aa:50070/tmp/锟斤拷锟斤拷路锟斤拷锟斤拷锟斤拷hdfs:
> //10.xx.xx.bb:54310/tmp/distcp_test14
>
> 12/08/28 23:32:31 INFO tools.DistCp: Input Options:
> DistCpOptions{atomicCommit=false, syncFolder=false,
> deleteMissing=false, ignoreFailures=true, maxMaps=14,
> sslConfigurationFile='null', copyStrategy='uniformsize',
> sourceFileListing=null, sourcePaths=[hftp://10.xx.xx.aa:50070/tmp/锟斤拷锟斤拷
> 路锟斤拷锟斤拷锟斤拷], targetPath=hdfs://10.xx.xx.bb:54310/tmp/distcp_test14}
>
> 12/08/28 23:32:33 INFO tools.DistCp: DistCp job log path: /tmp/distcp.log
>
> 12/08/28 23:32:34 WARN conf.Configuration: io.sort.mb is deprecated.
> Instead, use mapreduce.task.io.sort.mb
>
> 12/08/28 23:32:34 WARN conf.Configuration: io.sort.factor is
> deprecated. Instead, use mapreduce.task.io.sort.factor
>
> 12/08/28 23:32:34 WARN util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes
> where applicable
>
> 12/08/28 23:32:36 INFO mapreduce.JobSubmitter: number of splits:1
>
> 12/08/28 23:32:36 WARN conf.Configuration: mapred.jar is deprecated.
> Instead, use mapreduce.job.jar
>
> 12/08/28 23:32:36 WARN conf.Configuration:
> mapred.map.tasks.speculative.execution is deprecated. Instead, use
> mapreduce.map.speculative
>
> 12/08/28 23:32:36 WARN conf.Configuration: mapred.reduce.tasks is
> deprecated. Instead, use mapreduce.job.reduces
>
> 12/08/28 23:32:36 WARN conf.Configuration:
> mapred.mapoutput.value.class is deprecated. Instead, use
> mapreduce.map.output.value.class
>
> 12/08/28 23:32:36 WARN conf.Configuration: mapreduce.map.class is
> deprecated. Instead, use mapreduce.job.map.class
>
> 12/08/28 23:32:36 WARN conf.Configuration: mapred.job.name is
> deprecated. Instead, use mapreduce.job.name
>
> 12/08/28 23:32:36 WARN conf.Configuration: mapreduce.inputformat.class
> is deprecated. Instead, use mapreduce.job.inputformat.class
>
> 12/08/28 23:32:36 WARN conf.Configuration: mapred.output.dir is
> deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir
>
> 12/08/28 23:32:36 WARN conf.Configuration:
> mapreduce.outputformat.class is deprecated. Instead, use
> mapreduce.job.outputformat.class
>
> 12/08/28 23:32:36 WARN conf.Configuration: mapred.map.tasks is
> deprecated. Instead, use mapreduce.job.maps
>
> 12/08/28 23:32:36 WARN conf.Configuration: mapred.mapoutput.key.class
> is deprecated. Instead, use mapreduce.map.output.key.class
>
> 12/08/28 23:32:36 WARN conf.Configuration: mapred.working.dir is
> deprecated. Instead, use mapreduce.job.working.dir
>
> 12/08/28 23:32:37 INFO mapred.ResourceMgrDelegate: Submitted
> application application_1345831938927_0039 to ResourceManager at
> baby20/10.1.1.40:8040
>
> 12/08/28 23:32:37 INFO mapreduce.Job: The url to track the job:
> http://baby20:8088/proxy/application_1345831938927_0039/
>
> 12/08/28 23:32:37 INFO tools.DistCp: DistCp job-id: job_1345831938927_0039
>
> 12/08/28 23:32:37 INFO mapreduce.Job: Running job: job_1345831938927_0039
>
> 12/08/28 23:32:50 INFO mapreduce.Job: Job job_1345831938927_0039
> running in uber mode : false
>
> 12/08/28 23:32:50 INFO mapreduce.Job: map 0% reduce 0%
>
> 12/08/28 23:33:00 INFO mapreduce.Job: map 100% reduce 0%
>
> 12/08/28 23:33:00 INFO mapreduce.Job: Task Id :
> attempt_1345831938927_0039_m_000000_0, Status : FAILED
>
> Error: java.io.IOException: File copy failed: hftp://10.1.1.26:50070
> /tmp/锟斤拷锟斤拷路锟斤拷锟斤拷锟斤拷/part-r-00017 -->
> hdfs://10.1.1.40:54310/tmp/distcp_test14/part-r-00017
>
> at
> org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:262)
10mo. ANIVERSARIO DE LA CREACION DE LA UNIVERSIDAD DE LAS CIENCIAS INFORMATICAS...
CONECTADOS AL FUTURO, CONECTADOS A LA REVOLUCION
http://www.uci.cu
http://www.facebook.com/universidad.uci
http://www.flickr.com/photos/universidad_uci
+
Tao 2012-08-28, 16:30
