Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
HDFS >> mail # user >> Namenode hangs on startup


Copy link to this message
-
Namenode hangs on startup
Hi,

Apache Hadoop 0.20.205.

I'm trying to restart NN and it always hangs at the very beginning.
The only logs I've got are:

/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG:   host = host/ip
STARTUP_MSG:   args = []
STARTUP_MSG:   version = 0.20.205.0
STARTUP_MSG:   build https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20-security-205
-r 1179940; compiled by 'hortonfo' on Fri Oct  7 06:20:32 UTC 2011
************************************************************/
2012-07-02 01:33:01,281 INFO
org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from
hadoop-metrics2.properties
2012-07-02 01:33:01,290 INFO
org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
MetricsSystem,sub=Stats registered.
2012-07-02 01:33:01,292 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot
period at 10 second(s).
2012-07-02 01:33:01,292 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics
system started
2012-07-02 01:33:01,434 INFO
org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
ugi registered.
2012-07-02 01:33:01,436 WARN
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Source name ugi
already exists!
2012-07-02 01:33:01,441 INFO
org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
jvm registered.
2012-07-02 01:33:01,441 INFO
org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
NameNode registered.
2012-07-02 01:33:01,463 INFO org.apache.hadoop.hdfs.util.GSet: VM type
      = 64-bit
2012-07-02 01:33:01,463 INFO org.apache.hadoop.hdfs.util.GSet: 2% max
memory = 314.0275 MB
2012-07-02 01:33:01,463 INFO org.apache.hadoop.hdfs.util.GSet:
capacity      = 2^25 = 33554432 entries
2012-07-02 01:33:01,463 INFO org.apache.hadoop.hdfs.util.GSet:
recommended=33554432, actual=33554432
2012-07-02 01:33:01,546 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner=owner
2012-07-02 01:33:01,546 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
supergroup=supergroup
2012-07-02 01:33:01,546 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
isPermissionEnabled=true
2012-07-02 01:33:01,550 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
dfs.block.invalidate.limit=100
2012-07-02 01:33:01,550 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
isAccessTokenEnabled=false accessKeyUpdateInterval=0 min(s),
accessTokenLifetime=0 min(s)
2012-07-02 01:33:01,787 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Registered
FSNamesystemStateMBean and NameNodeMXBean
2012-07-02 01:33:01,802 INFO
org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names
occuring more than 10 times
2012-07-02 01:33:01,811 INFO
org.apache.hadoop.hdfs.server.common.Storage: Number of files = 17032
2012-07-02 01:33:02,406 INFO
org.apache.hadoop.hdfs.server.common.Storage: Number of files under
construction = 0
2012-07-02 01:33:02,406 INFO
org.apache.hadoop.hdfs.server.common.Storage: Image file of size
2553316 loaded in 0 seconds.
2012-07-02 01:33:02,410 INFO
org.apache.hadoop.hdfs.server.common.Storage: Edits file
/apr/hdfs/name/current/edits of size 498 edits # 7 loaded in 0
seconds.

===================================
It hangs thereafter.... I wonder if anybody has seen this before?

Some background: I shut down DFS and MR while there were still jobs
running. Some MR jobs were hanging, so I manually killed the children
JVMs after the shutdown. Not sure how such actions would affect NN
startup.

Any help would be appreciated.

Thanks,
James
+
Harsh J 2012-07-02, 10:21
+
Jianhui Zhang 2012-07-02, 19:04
+
Harsh J 2012-07-02, 19:07
+
Jianhui Zhang 2012-07-02, 19:32
+
David Rosenstrauch 2012-07-02, 19:56
+
Jianhui Zhang 2012-07-02, 22:30
+
Harsh J 2012-07-03, 02:21
+
Mohammad Tariq 2012-07-02, 09:32
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB