Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce, mail # user - HDFS Startup Failure due to dfs.namenode.rpc-address and Shared Edits Directory


Copy link to this message
-
RE: HDFS Startup Failure due to dfs.namenode.rpc-address and Shared Edits Directory
Smith, Joshua D. 2013-08-27, 16:16
That fixed it. I was assuming that nn1 and nn2 were hostnames and not IDs.

Once I replaced the value with nn1,nn2, everything started to make sense.

Thank you Azurry and Jitendra. Much appreciated!

Josh

From: Jitendra Yadav [mailto:[EMAIL PROTECTED]]
Sent: Tuesday, August 27, 2013 11:54 AM
To: [EMAIL PROTECTED]
Subject: Re: HDFS Startup Failure due to dfs.namenode.rpc-address and Shared Edits Directory

Hi,

Please follow the HA configuration steps available at below link.

http://hadoop.apache.org/docs/r2.1.0-beta/hadoop-yarn/hadoop-yarn-site/HDFSHighAvailabilityWithQJM.html
"dfs.ha.namenodes.[nameservice ID] - unique identifiers for each NameNode in the nameservice

Configure with a list of comma-separated NameNode IDs. This will be used by DataNodes to determine all the NameNodes in the cluster. For example, if you used "mycluster" as the nameservice ID previously, and you wanted to use "nn1" and "nn2" as the individual IDs of the NameNodes, you would configure this as such:

<property>
  <name>dfs.ha.namenodes.mycluster</name>
  <value>nn1,nn2</value>
</property>"

Thanks
On Tue, Aug 27, 2013 at 9:08 PM, Smith, Joshua D. <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:
nn.domain is a place holder for the actual fully qualified hostname of my NameNode
snn.domain is a place holder for the actual fully qualified hostname of my StandbyNameNode.

Of course both the NameNode and the StandbyNameNode are running exactly the same software with the same configuration since this is YARN. I'm not running and SecondaryName node.

The actual fully qualified hostnames are on another network and my customer is sensitive about privacy, so that's why I didn't post the actual values.

So, I think I have the equivalent of nn1,nn2 do I not?

From: Azuryy Yu [mailto:[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>]
Sent: Tuesday, August 27, 2013 11:32 AM

To: [EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>
Subject: RE: HDFS Startup Failure due to dfs.namenode.rpc-address and Shared Edits Directory
dfs.ha.namenodes.mycluster
nn.domain,snn.domain

it should be:
dfs.ha.namenodes.mycluster
nn1,nn2
On Aug 27, 2013 11:22 PM, "Smith, Joshua D." <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:
Harsh-

Here are all of the other values that I have configured.

hdfs-site.xml
-----------------

dfs.webhdfs.enabled
true

dfs.client.failover.proxy.provider.mycluster
org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider

dfs.ha.automatic-falover.enabled
true

ha.zookeeper.quorum
nn.domain:2181,snn.domain:2181,jt.domain:2181

dfs.journalnode.edits.dir
/opt/hdfs/data1/dfs/jn

dfs.namenode.shared.edits.dir
qjournal://nn.domain:8485;snn.domain:8485;jt.domain:8485/mycluster

dfs.nameservices
mycluster

dfs.ha.namenodes.mycluster
nn.domain,snn.domain

dfs.namenode.rpc-address.mycluster.nn1
nn.domain:8020

dfs.namenode.rpc-address.mycluster.nn2
snn.domain:8020

dfs.namenode.http-address.mycluster.nn1
nn.domain:50070

dfs.namenode.http-address.mycluster.nn2
snn.domain:50070

dfs.name.dir
/var/lib/hadoop-hdfs/cache/hdfs/dfs/name
core-site.xml
----------------
fs.trash.interval
1440

fs.trash.checkpoint.interval
1440

fs.defaultFS
hdfs://mycluster

dfs.datanode.data.dir
/hdfs/data1,/hdfs/data2,/hdfs/data3,/hdfs/data4,/hdfs/data5,/hdfs/data6,/hdfs/data7
mapred-site.xml
----------------------
mapreduce.framework.name<http://mapreduce.framework.name/>
yarn

mapreduce.jobhistory.address
jt.domain:10020

mapreduce.jobhistory.webapp.address
jt.domain:19888
yarn-site.xml
-------------------
yarn.nodemanager.aux-service
mapreduce.shuffle

yarn.nodemanager.aux-services.mapreduce.shuffle.class
org.apache.hadoop.mapred.ShuffleHandler

yarn.log-aggregation-enable
true

yarn.nodemanager.remote-app-log-dir
/var/log/hadoop-yarn/apps

yarn.application.classpath
$HADOOP_CONF_DIR,$HADOOP_COMMON_HOME/*,$HADOOP_COMMON_HOME/lib/*,$HADOOP_HDFS_HOME/*,$HADOOP_HDFS_HOME/lib /*,$HADOOP_MAPRED_HOME/*,$HADOOP_MAPRED_HOME/lib/*,$YARN_HOME/*,$YARN_HOME/lib/*

yarn.resourcemanager.resource-tracker.address
jt.domain:8031

yarn.resourcemanager.address
jt.domain:8032

yarn.resourcemanager.scheduler.address
jt.domain:8030

yarn.resourcemanager.admin.address
jt.domain:8033

yarn.reesourcemanager.webapp.address
jt.domain:8088
These are the only interesting entries in my HDFS log file when I try to start the NameNode with "service hadoop-hdfs-namenode start".

WARN org.apache.hadoop.hdfs.server.common.Util: Path /var/lib/hadoop-hdfs/cache/hdfs/dfs/name should be specified as a URI in configuration files. Please update hdfs configuration.
WARN org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one image storage directory (dfs.namenode.name.dir) configured. Beware of data loss due to lack of redundant storage directories!
INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled: false
WARN org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Configured NNs:
((there's a blank line here implying no configured NameNodes!))
ERROR org.apache.hadoop.hdfs.server.namenode.FSNamesystem: FSNamesystem initialization failed.
Java.io.IOException: Invalid configuration: a shared edits dir must not be specified if HA is not enabled.
FATAL org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode join
Java.io.IOException: Invalid configuration: a shared edits dir must not be specified if HA is not enabled.

I don't like the blank line for Configured NNs. Not sure why it's not finding them.

If I try the command "hdfs zkfc -formatZK" I get the following:
Exception in thread "main" org.apache.hadoop.HadoopIllegalArgumentException: HA is not enabled for this namenode.

From: Smith, Joshua D. [mailto:[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>]
Sent: Tuesday, August 27, 2013 7:17 AM
To: [EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>
Subject: RE: HDFS Startup Failure due to dfs.namenode.rpc-address and Shared Edits Directory

Harsh-

Yes, I intend to use