Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HDFS, mail # user - Datanode support different Namespace


Copy link to this message
-
Datanode support different Namespace
Bing Jiang 2013-07-04, 07:12
Hi, all

We try to use hadoop-2.0.5-alpha, using two namespaces, one is for hbase
cluster, and the other one is for common use.At the same time, we use
Quorum Journal policy as HA.

GS-CIX-SEV0001, GS-CIX-SEV0002,  namenodes in hbasecluster  namespace

GS-CIX-SEV0003, GS-CIX-SEV0004, namenodes in commoncluster namespace.

GS-CIX-SEV0001~GS-CIX-SEV0008 , 8 machines used as Datanode

After launching the hdfs cluster all, there is something which makes me
confused, that  each namespace has half of the datanodes.

NameNode 'GS-CIX-SEV0004:9100'

Started:        Thu Jul 04 10:28:00 CST 2013
Version:        2.0.5-alpha, 1488459
Compiled:       2013-06-01T04:05Z by jenkins from branch-2.0.5-alpha
Cluster ID:     CID-15c48d78-2137-4c6e-aacf-0edbf2bb3db7
Block Pool ID:  BP-1792015895-10.100.2.3-1372904504940
Browse the filesystem
NameNode Logs
Go back to DFS home
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Live Datanodes : 4
                Last     Admin    Configured  Used Non DFS  Remaining
Used      Used      Remaining          Block     Block Pool      Failed
     Node      Contact   State     Capacity   (GB)   Used     (GB)
(%)       (%)          (%)    Blocks   Pool      Used (%)>      Volumes
                                     (GB)
(GB)                                                  Used (GB)     Blocks
GS-CIX-SEV0001 1       In Service      888.07 0.00   116.04    772.03 0.00
┌────────────┐     86.93 0           0.00           0.00            0

└────────────┘
GS-CIX-SEV0002 1       In Service      888.07 0.00   135.50    752.57 0.00
┌────────────┐     84.74 0           0.00           0.00            0

└────────────┘
GS-CIX-SEV0005 1       In Service      888.07 0.00    97.61    790.46 0.00
┌────────────┐     89.01 0           0.00           0.00            0

└────────────┘
GS-CIX-SEV0006 1       In Service      888.07 0.00   122.30    765.77 0.00
┌────────────┐     86.23 0           0.00           0.00            0

└────────────┘
Another Namespace's NameNode:

NameNode 'GS-CIX-SEV0001:9100'

Started:        Thu Jul 04 10:19:03 CST 2013
Version:        2.0.5-alpha, 1488459
Compiled:       2013-06-01T04:05Z by jenkins from branch-2.0.5-alpha
Cluster ID:     CID-1a53483d-000e-4726-aef1-f500bedb1df6
Block Pool ID:  BP-1142418822-10.100.2.1-1372904314309
Browse the filesystem
NameNode Logs
Go back to DFS home
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Live Datanodes : 4
                Last     Admin    Configured  Used Non DFS  Remaining
Used      Used      Remaining          Block     Block Pool      Failed
     Node      Contact   State     Capacity   (GB)   Used     (GB)
(%)       (%)          (%)    Blocks   Pool      Used (%)>      Volumes
                                     (GB)
(GB)                                                  Used (GB)     Blocks
GS-CIX-SEV0003 0       In Service      888.07 0.00   150.54    737.53 0.00
┌────────────┐     83.05 0           0.00           0.00            0

└────────────┘
GS-CIX-SEV0004 0       In Service      888.07 0.00   177.22    710.85 0.00
┌────────────┐     80.04 0           0.00           0.00            0

└────────────┘
GS-CIX-SEV0007 0       In Service      888.07 0.00    62.91    825.16 0.00
┌────────────┐     92.92 0           0.00           0.00            0

└────────────┘
GS-CIX-SEV0008 0       In Service      888.07 0.00   125.25    762.82 0.00
┌────────────┐     85.90 0           0.00           0.00            0

And check the DN(GS-CIX-SEV0001)'s log, it prints like this:
2013-07-04 10:34:51,699 FATAL
org.apache.hadoop.hdfs.server.datanode.DataNode: Initialization failed for
block pool Block pool BP-1142418822-10.100.2.1-1372904314309 (storage id
DS-1677272131-10.100.2.1-50010-1372905291690) service to GS-CIX-SEV0001/
10.100.2.1:9100
java.io.IOException: Inconsistent storage IDs. Name-node returned
DS811369792. Expecting DS-1677272131-10.100.2.1-50010-1372905291690
        at
org.apache.hadoop.hdfs.server.datanode.DataNode.bpRegistrationSucceeded(DataNode.java:731)
        at
org.apache.hadoop.hdfs.server.datanode.BPOfferService.registrationSucceeded(BPOfferService.java:308)
        at
org.apache.hadoop.hdfs.server.datanode.BPServiceActor.register(BPServiceActor.java:632)
        at
org.apache.hadoop.hdfs.server.datanode.BPServiceActor.connectToNNAndHandshake(BPServiceActor.java:225)
        at
org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:664)
        at java.lang.Thread.run(Thread.java:662)

It is proved that one datanode has been required to attached to only one
namespace?

Any views about it will be thankful.

Regards~
Bing Jiang
Tel:(86)134-2619-1361
weibo: http://weibo.com/jiangbinglover
BLOG: http://blog.sina.com.cn/jiangbinglover
National Research Center for Intelligent Computing Systems
Institute of Computing technology
Graduate University of Chinese Academy of Science