Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce >> mail # user >> There are 2 datanode(s) running and 2 node(s) are excluded in this operation.


Copy link to this message
-
Re: There are 2 datanode(s) running and 2 node(s) are excluded in this operation.
I am copying from Site1 to Site2 in Amazon EC2. Each site has mounted 1
hadoop instance with 1 NN and 2 DNs. I think my problem is something
related distcp command being unable to query one of the HDFS' clusters'
DNs like Harsh told. Amazon EC2 use private IPs to communicate between
hosts in the same site and public IPs to connect between sites. I am
using distcp to copy big data (256MB each file) between different sites
using the public IP:
hadoop distcp hdfs://publicIP1:9000/wiki hdfs://publicIP2:9000/wiki

Checking the logs (hadoop-ubuntu-namenode-ip-XX-XXX-XXX-94.log), I also
got this problem  but I don't know if it is related:
2013-08-28 07:40:17,537 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 3 on 9000, call
org.apache.hadoop.hdfs.protocol.ClientProtocol.abandonBlock from
XX.XXX.XXX.150:51844: error:
org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: No lease
on /wiki/.distcp.tmp.attempt_1377674569447_0001_m_000000_1: File does
not exist. H
older DFSClient_attempt_1377674569447_0001_m_000000_1_-1864394963_1 does
not have any open files.
org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: No lease
on /wiki/.distcp.tmp.attempt_1377674569447_0001_m_000000_1: File does
not exist. Holder DFSClient_
attempt_1377674569447_0001_m_000000_1_-1864394963_1 does not have any
open files.
Here is the report:

Site1
$ hdfs dfsadmin -report
13/08/28 11:17:52 WARN util.NativeCodeLoader: Unable to load
native-hadoop library for your platform... using builtin-java classes
where applicable
Configured Capacity: 25365413888 (23.62 GB)
Present Capacity: 20488749056 (19.08 GB)
DFS Remaining: 20487512064 (19.08 GB)
DFS Used: 1236992 (1.18 MB)
DFS Used%: 0.01%
Under replicated blocks: 0
Blocks with corrupt replicas: 0
Missing blocks: 0

-------------------------------------------------
Datanodes available: 2 (2 total, 0 dead)

Live datanodes:
Name: XX.XXX.XXX.243:50010 (ip-XX-XXX-XXX-243.eu-west-1.compute.internal)
Hostname: ip-XX-XXX-XXX-243.eu-west-1.compute.internal
Decommission Status : Normal
Configured Capacity: 12682706944 (11.81 GB)
DFS Used: 618496 (604 KB)
Non DFS Used: 2489556992 (2.32 GB)
DFS Remaining: 10192531456 (9.49 GB)
DFS Used%: 0.00%
DFS Remaining%: 80.37%
Last contact: Wed Aug 28 11:17:51 UTC 2013
Name: XX.XXX.XXX.58:50010 (ip-XX-XXX-XXX-58.eu-west-1.compute.internal)
Hostname: ip-XX-XXX-XXX-58.eu-west-1.compute.internal
Decommission Status : Normal
Configured Capacity: 12682706944 (11.81 GB)
DFS Used: 618496 (604 KB)
Non DFS Used: 2387107840 (2.22 GB)
DFS Remaining: 10294980608 (9.59 GB)
DFS Used%: 0.00%
DFS Remaining%: 81.17%
Last contact: Wed Aug 28 11:17:51 UTC 2013
Site2:
$ hdfs dfsadmin -report
13/08/28 11:17:55 WARN util.NativeCodeLoader: Unable to load
native-hadoop library for your platform... using builtin-java classes
where applicable
Configured Capacity: 25365413888 (23.62 GB)
Present Capacity: 20593844224 (19.18 GB)
DFS Remaining: 20593770496 (19.18 GB)
DFS Used: 73728 (72 KB)
DFS Used%: 0.00%
Under replicated blocks: 0
Blocks with corrupt replicas: 0
Missing blocks: 0

-------------------------------------------------
Datanodes available: 2 (2 total, 0 dead)

Live datanodes:
Name: XX.XXX.XXX.95:50010 (ip-XX-XXX-XXX-95.ap-southeast-1.compute.internal)
Hostname: ip-XX-XXX-XXX-95.ap-southeast-1.compute.internal
Decommission Status : Normal
Configured Capacity: 12682706944 (11.81 GB)
DFS Used: 36864 (36 KB)
Non DFS Used: 2385768448 (2.22 GB)
DFS Remaining: 10296901632 (9.59 GB)
DFS Used%: 0.00%
DFS Remaining%: 81.19%
Last contact: Wed Aug 28 11:17:56 UTC 2013
Name: XX.XXX.XXX.96:50010 (ip-XX-XXX-XXX-96.ap-southeast-1.compute.internal)
Hostname: ip-XX-XXX-XXX-96.ap-southeast-1.compute.internal
Decommission Status : Normal
Configured Capacity: 12682706944 (11.81 GB)
DFS Used: 36864 (36 KB)
Non DFS Used: 2385801216 (2.22 GB)
DFS Remaining: 10296868864 (9.59 GB)
DFS Used%: 0.00%
DFS Remaining%: 81.19%
Last contact: Wed Aug 28 11:17:56 UTC 2013

Any suggestion to fix this problem?
On 08/28/2013 12:09 PM, Jitendra Yadav wrote:
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB