Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
HDFS >> mail # user >> Namenode hangs on startup


+
Jianhui Zhang 2012-07-02, 09:04
Copy link to this message
-
Re: Namenode hangs on startup
Jianhui,

Can you pastebin.com the output of your "jstack <NN PID>" command
after its hung, and pass us the paste link please? It looks to me like
it may have just been merging/saving the image, and that may be slow
but it depends on how long did you have to wait around to see NN
resume and begin properly.

On Mon, Jul 2, 2012 at 2:34 PM, Jianhui Zhang <[EMAIL PROTECTED]> wrote:
> Hi,
>
> Apache Hadoop 0.20.205.
>
> I'm trying to restart NN and it always hangs at the very beginning.
> The only logs I've got are:
>
> /************************************************************
> STARTUP_MSG: Starting NameNode
> STARTUP_MSG:   host = host/ip
> STARTUP_MSG:   args = []
> STARTUP_MSG:   version = 0.20.205.0
> STARTUP_MSG:   build > https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20-security-205
> -r 1179940; compiled by 'hortonfo' on Fri Oct  7 06:20:32 UTC 2011
> ************************************************************/
> 2012-07-02 01:33:01,281 INFO
> org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from
> hadoop-metrics2.properties
> 2012-07-02 01:33:01,290 INFO
> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
> MetricsSystem,sub=Stats registered.
> 2012-07-02 01:33:01,292 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot
> period at 10 second(s).
> 2012-07-02 01:33:01,292 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics
> system started
> 2012-07-02 01:33:01,434 INFO
> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
> ugi registered.
> 2012-07-02 01:33:01,436 WARN
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Source name ugi
> already exists!
> 2012-07-02 01:33:01,441 INFO
> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
> jvm registered.
> 2012-07-02 01:33:01,441 INFO
> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
> NameNode registered.
> 2012-07-02 01:33:01,463 INFO org.apache.hadoop.hdfs.util.GSet: VM type
>       = 64-bit
> 2012-07-02 01:33:01,463 INFO org.apache.hadoop.hdfs.util.GSet: 2% max
> memory = 314.0275 MB
> 2012-07-02 01:33:01,463 INFO org.apache.hadoop.hdfs.util.GSet:
> capacity      = 2^25 = 33554432 entries
> 2012-07-02 01:33:01,463 INFO org.apache.hadoop.hdfs.util.GSet:
> recommended=33554432, actual=33554432
> 2012-07-02 01:33:01,546 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner=owner
> 2012-07-02 01:33:01,546 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> supergroup=supergroup
> 2012-07-02 01:33:01,546 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> isPermissionEnabled=true
> 2012-07-02 01:33:01,550 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> dfs.block.invalidate.limit=100
> 2012-07-02 01:33:01,550 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> isAccessTokenEnabled=false accessKeyUpdateInterval=0 min(s),
> accessTokenLifetime=0 min(s)
> 2012-07-02 01:33:01,787 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Registered
> FSNamesystemStateMBean and NameNodeMXBean
> 2012-07-02 01:33:01,802 INFO
> org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names
> occuring more than 10 times
> 2012-07-02 01:33:01,811 INFO
> org.apache.hadoop.hdfs.server.common.Storage: Number of files = 17032
> 2012-07-02 01:33:02,406 INFO
> org.apache.hadoop.hdfs.server.common.Storage: Number of files under
> construction = 0
> 2012-07-02 01:33:02,406 INFO
> org.apache.hadoop.hdfs.server.common.Storage: Image file of size
> 2553316 loaded in 0 seconds.
> 2012-07-02 01:33:02,410 INFO
> org.apache.hadoop.hdfs.server.common.Storage: Edits file
> /apr/hdfs/name/current/edits of size 498 edits # 7 loaded in 0
> seconds.
>
> ===================================>
> It hangs thereafter.... I wonder if anybody has seen this before?
>
> Some background: I shut down DFS and MR while there were still jobs
> running. Some MR jobs were hanging, so I manually killed the children

Harsh J
+
Jianhui Zhang 2012-07-02, 19:04
+
Harsh J 2012-07-02, 19:07
+
Jianhui Zhang 2012-07-02, 19:32
+
David Rosenstrauch 2012-07-02, 19:56
+
Jianhui Zhang 2012-07-02, 22:30
+
Harsh J 2012-07-03, 02:21
+
Mohammad Tariq 2012-07-02, 09:32
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB