Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HDFS, mail # user - Namenode hangs on startup


Copy link to this message
-
Namenode hangs on startup
Jianhui Zhang 2012-07-02, 09:04
Hi,

Apache Hadoop 0.20.205.

I'm trying to restart NN and it always hangs at the very beginning.
The only logs I've got are:

/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG:   host = host/ip
STARTUP_MSG:   args = []
STARTUP_MSG:   version = 0.20.205.0
STARTUP_MSG:   build https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20-security-205
-r 1179940; compiled by 'hortonfo' on Fri Oct  7 06:20:32 UTC 2011
************************************************************/
2012-07-02 01:33:01,281 INFO
org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from
hadoop-metrics2.properties
2012-07-02 01:33:01,290 INFO
org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
MetricsSystem,sub=Stats registered.
2012-07-02 01:33:01,292 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot
period at 10 second(s).
2012-07-02 01:33:01,292 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics
system started
2012-07-02 01:33:01,434 INFO
org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
ugi registered.
2012-07-02 01:33:01,436 WARN
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Source name ugi
already exists!
2012-07-02 01:33:01,441 INFO
org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
jvm registered.
2012-07-02 01:33:01,441 INFO
org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
NameNode registered.
2012-07-02 01:33:01,463 INFO org.apache.hadoop.hdfs.util.GSet: VM type
      = 64-bit
2012-07-02 01:33:01,463 INFO org.apache.hadoop.hdfs.util.GSet: 2% max
memory = 314.0275 MB
2012-07-02 01:33:01,463 INFO org.apache.hadoop.hdfs.util.GSet:
capacity      = 2^25 = 33554432 entries
2012-07-02 01:33:01,463 INFO org.apache.hadoop.hdfs.util.GSet:
recommended=33554432, actual=33554432
2012-07-02 01:33:01,546 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner=owner
2012-07-02 01:33:01,546 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
supergroup=supergroup
2012-07-02 01:33:01,546 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
isPermissionEnabled=true
2012-07-02 01:33:01,550 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
dfs.block.invalidate.limit=100
2012-07-02 01:33:01,550 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
isAccessTokenEnabled=false accessKeyUpdateInterval=0 min(s),
accessTokenLifetime=0 min(s)
2012-07-02 01:33:01,787 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Registered
FSNamesystemStateMBean and NameNodeMXBean
2012-07-02 01:33:01,802 INFO
org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names
occuring more than 10 times
2012-07-02 01:33:01,811 INFO
org.apache.hadoop.hdfs.server.common.Storage: Number of files = 17032
2012-07-02 01:33:02,406 INFO
org.apache.hadoop.hdfs.server.common.Storage: Number of files under
construction = 0
2012-07-02 01:33:02,406 INFO
org.apache.hadoop.hdfs.server.common.Storage: Image file of size
2553316 loaded in 0 seconds.
2012-07-02 01:33:02,410 INFO
org.apache.hadoop.hdfs.server.common.Storage: Edits file
/apr/hdfs/name/current/edits of size 498 edits # 7 loaded in 0
seconds.

===================================
It hangs thereafter.... I wonder if anybody has seen this before?

Some background: I shut down DFS and MR while there were still jobs
running. Some MR jobs were hanging, so I manually killed the children
JVMs after the shutdown. Not sure how such actions would affect NN
startup.

Any help would be appreciated.

Thanks,
James