Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
MapReduce >> mail # user >> HDFS Startup Failure due to dfs.namenode.rpc-address and Shared Edits Directory


+
Smith, Joshua D. 2013-08-26, 19:18
+
Harsh J 2013-08-26, 22:11
+
Smith, Joshua D. 2013-08-27, 11:17
+
Smith, Joshua D. 2013-08-27, 15:38
Copy link to this message
-
Re: HDFS Startup Failure due to dfs.namenode.rpc-address and Shared Edits Directory
Hi,

Please follow the HA configuration steps available at below link.

http://hadoop.apache.org/docs/r2.1.0-beta/hadoop-yarn/hadoop-yarn-site/HDFSHighAvailabilityWithQJM.html
*"dfs.ha.namenodes.[nameservice ID] - unique identifiers for each NameNode
in the nameservice *

*Configure with a list of comma-separated NameNode IDs. This will be used
by DataNodes to determine all the NameNodes in the cluster. For example, if
you used "mycluster" as the nameservice ID previously, and you wanted to
use "nn1" and "nn2" as the individual IDs of the NameNodes, you would
configure this as such:*

*<property>
  <name>dfs.ha.namenodes.mycluster</name>
  <value>nn1,nn2</value>
</property>"*

Thanks
On Tue, Aug 27, 2013 at 9:08 PM, Smith, Joshua D.
<[EMAIL PROTECTED]>wrote:

>  nn.domain is a place holder for the actual fully qualified hostname of
> my NameNode****
>
> snn.domain is a place holder for the actual fully qualified hostname of my
> StandbyNameNode.****
>
> ** **
>
> Of course both the NameNode and the StandbyNameNode are running exactly
> the same software with the same configuration since this is YARN. I’m not
> running and SecondaryName node.****
>
> ** **
>
> The actual fully qualified hostnames are on another network and my
> customer is sensitive about privacy, so that’s why I didn’t post the actual
> values.****
>
> ** **
>
> So, I think I have the equivalent of nn1,nn2 do I not?****
>
> ** **
>
> *From:* Azuryy Yu [mailto:[EMAIL PROTECTED]]
> *Sent:* Tuesday, August 27, 2013 11:32 AM
>
> *To:* [EMAIL PROTECTED]
> *Subject:* RE: HDFS Startup Failure due to dfs.namenode.rpc-address and
> Shared Edits Directory****
>
>  ** **
>
> dfs.ha.namenodes.mycluster
> nn.domain,snn.domain****
>
> it should be:
> dfs.ha.namenodes.mycluster
> nn1,nn2****
>
> On Aug 27, 2013 11:22 PM, "Smith, Joshua D." <[EMAIL PROTECTED]>
> wrote:****
>
> Harsh-
>
> Here are all of the other values that I have configured.
>
> hdfs-site.xml
> -----------------
>
> dfs.webhdfs.enabled
> true
>
> dfs.client.failover.proxy.provider.mycluster
> org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider
>
> dfs.ha.automatic-falover.enabled
> true
>
> ha.zookeeper.quorum
> nn.domain:2181,snn.domain:2181,jt.domain:2181
>
> dfs.journalnode.edits.dir
> /opt/hdfs/data1/dfs/jn
>
> dfs.namenode.shared.edits.dir
> qjournal://nn.domain:8485;snn.domain:8485;jt.domain:8485/mycluster
>
> dfs.nameservices
> mycluster
>
> dfs.ha.namenodes.mycluster
> nn.domain,snn.domain
>
> dfs.namenode.rpc-address.mycluster.nn1
> nn.domain:8020
>
> dfs.namenode.rpc-address.mycluster.nn2
> snn.domain:8020
>
> dfs.namenode.http-address.mycluster.nn1
> nn.domain:50070
>
> dfs.namenode.http-address.mycluster.nn2
> snn.domain:50070
>
> dfs.name.dir
> /var/lib/hadoop-hdfs/cache/hdfs/dfs/name
>
>
> core-site.xml
> ----------------
> fs.trash.interval
> 1440
>
> fs.trash.checkpoint.interval
> 1440
>
> fs.defaultFS
> hdfs://mycluster
>
> dfs.datanode.data.dir
>
> /hdfs/data1,/hdfs/data2,/hdfs/data3,/hdfs/data4,/hdfs/data5,/hdfs/data6,/hdfs/data7
>
>
> mapred-site.xml
> ----------------------
> mapreduce.framework.name
> yarn
>
> mapreduce.jobhistory.address
> jt.domain:10020
>
> mapreduce.jobhistory.webapp.address
> jt.domain:19888
>
>
> yarn-site.xml
> -------------------
> yarn.nodemanager.aux-service
> mapreduce.shuffle
>
> yarn.nodemanager.aux-services.mapreduce.shuffle.class
> org.apache.hadoop.mapred.ShuffleHandler
>
> yarn.log-aggregation-enable
> true
>
> yarn.nodemanager.remote-app-log-dir
> /var/log/hadoop-yarn/apps
>
> yarn.application.classpath
> $HADOOP_CONF_DIR,$HADOOP_COMMON_HOME/*,$HADOOP_COMMON_HOME/lib/*,$HADOOP_HDFS_HOME/*,$HADOOP_HDFS_HOME/lib
> /*,$HADOOP_MAPRED_HOME/*,$HADOOP_MAPRED_HOME/lib/*,$YARN_HOME/*,$YARN_HOME/lib/*
>
> yarn.resourcemanager.resource-tracker.address
> jt.domain:8031
>
> yarn.resourcemanager.address
> jt.domain:8032
>
> yarn.resourcemanager.scheduler.address
> jt.domain:8030
>
> yarn.resourcemanager.admin.address
> jt.domain:8033
+
Smith, Joshua D. 2013-08-27, 16:16
+
Azuryy Yu 2013-08-27, 16:00
+
Smith, Joshua D. 2013-08-27, 15:21