Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce >> mail # user >> How to use Hadoop2 HA's logical name URL?


Copy link to this message
-
How to use Hadoop2 HA's logical name URL?
Hi

I have setting up Hadoop 2.2.0 HA cluster following :
http://hadoop.apache.org/docs/r2.2.0/hadoop-yarn/hadoop-yarn-site/HDFSHighAvailabilityWithQJM.html#Configuration_details

And I can check both the active and standby namenode with WEB interface.

While, it seems that the logical name could not be used to access HDFS ?

I have following settings related to HA :

In core-site.xml:

    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://public-cluster</value>
    </property>
    <property>
        <name>dfs.ha.fencing.methods</name>
        <value>sshfence</value>
    </property>

    <property>
        <name>dfs.ha.fencing.ssh.private-key-files</name>
        <value>/root/.ssh/id_rsa</value>
    </property>

    <property>
        <name>dfs.ha.fencing.ssh.connect-timeout</name>
        <value>30000</value>
    </property>

And in hdfs-site.xml:

<property>
  <name>dfs.nameservices</name>
  <value>public-cluster</value>
</property>

<property>
  <name>dfs.ha.namenodes.public-cluster</name>
  <value>nn1,nn2</value>
</property>

<property>
  <name>dfs.namenode.rpc-address.public-cluster.nn1</name>
  <value>10.0.2.31:8020</value>
</property>
<property>
  <name>dfs.namenode.rpc-address.public-cluster.nn2</name>
  <value>10.0.2.32:8020</value>
</property>
<property>
  <name>dfs.namenode.http-address.public-cluster.nn1</name>
  <value>10.0.2.31:50070</value>
</property>
<property>
  <name>dfs.namenode.http-address.public-cluster.nn2</name>
  <value>10.0.2.32:50070</value>
</property>
<property>
  <name>dfs.namenode.shared.edits.dir</name>
  <value>qjournal://10.0.0.144:8485;10.0.0.145:8485;10.0.0.146:8485/public-cluster</value>
</property>
<property>
  <name>dfs.journalnode.edits.dir</name>
  <value>/mnt/DP_disk1/hadoop2/hdfs/jn</value>
</property>

<property>
  <name>dfs.client.failover.proxy.provider.mycluster</name>
  <value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
</property>

-------------------

And then :

./bin/hdfs dfs -fs hdfs://public-cluster -ls /
-ls: java.net.UnknownHostException: public-cluster
Usage: hadoop fs [generic options] -ls [-d] [-h] [-R] [<path> ...]

While if I use the active namenode's URL, it works:

./bin/hdfs dfs -fs hdfs://10.0.2.31:8020 -ls /
Found 1 items
drwxr-xr-x   - root supergroup          0 2013-10-24 14:30 /tmp

However, shouldn't this hdfs://public-cluster kind of thing works? Anything that I might miss to make it work? Thanks!

Best Regards,
Raymond Liu