Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # user >> Error using hadoop distcp


Copy link to this message
-
Re: Error using hadoop distcp
Hi praveenesh
             Can you try repeating the distcp using IP instead of host name. From the error looks like an RPC exception not able to identify the host, so I believe it can't be due to not setting a password less ssh. Just try it out.
Regards
Bejoy K S

-----Original Message-----
From: trang van anh <[EMAIL PROTECTED]>
Date: Wed, 05 Oct 2011 14:06:11
To: <[EMAIL PROTECTED]>
Reply-To: [EMAIL PROTECTED]
Subject: Re: Error using hadoop distcp

which  host run the task that throws the exception ? ensure that each
data node know another data nodes in hadoop cluster-> add "ub16" entry
in /etc/hosts on where the task running.
On 10/5/2011 12:15 PM, praveenesh kumar wrote:
> I am trying to use distcp to copy a file from one HDFS to another.
>
> But while copying I am getting the following exception :
>
> hadoop distcp hdfs://ub13:54310/user/hadoop/weblog
> hdfs://ub16:54310/user/hadoop/weblog
>
> 11/10/05 10:41:01 INFO mapred.JobClient: Task Id :
> attempt_201110031447_0005_m_000007_0, Status : FAILED
> java.net.UnknownHostException: unknown host: ub16
>          at org.apache.hadoop.ipc.Client$Connection.<init>(Client.java:195)
>          at org.apache.hadoop.ipc.Client.getConnection(Client.java:850)
>          at org.apache.hadoop.ipc.Client.call(Client.java:720)
>          at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
>          at $Proxy1.getProtocolVersion(Unknown Source)
>          at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:359)
>          at
> org.apache.hadoop.hdfs.DFSClient.createRPCNamenode(DFSClient.java:113)
>          at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:215)
>          at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:177)
>          at
> org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:82)
>          at
> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1378)
>          at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:66)
>          at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1390)
>          at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:196)
>          at org.apache.hadoop.fs.Path.getFileSystem(Path.java:175)
>          at
> org.apache.hadoop.mapred.FileOutputCommitter.setupJob(FileOutputCommitter.java:48)
>          at
> org.apache.hadoop.mapred.OutputCommitter.setupJob(OutputCommitter.java:124)
>          at org.apache.hadoop.mapred.Task.runJobSetupTask(Task.java:835)
>          at org.apache.hadoop.mapred.MapTask.run(MapTask.java:296)
>          at org.apache.hadoop.mapred.Child.main(Child.java:170)
>
> Its saying its not finding ub16. But the entry is there in /etc/hosts files.
> I am able to ssh both the machines. Do I need password less ssh between
> these two NNs ?
> What can be the issue ? Any thing I am missing before using distcp ?
>
> Thanks,
> Praveenesh
>

NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB