Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HDFS, mail # user - HDFS edit log NPE


Copy link to this message
-
Re: HDFS edit log NPE
Tsz Wo Sze 2013-06-05, 07:41
Is it an operation error on upgrade since the edit is non-empty?  The
original image and edit should be still available.  If it is the case, I suggest to start NN with 1.0.4 so that the edit becomes empty, and then try upgrade again.
> Recent opcode offsets: 5 14

BTW, opcode 5 is OP_DATANODE_ADD which was deprecated long time ago.  It
seems that v1.1.2 cannot understand v1.0.4 edit.  Otherwise, the
edit log is corrupted.

Hope it helps.
Tsz-Wo
________________________________
 From: Robert Dyer <[EMAIL PROTECTED]>
To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>
Sent: Tuesday, June 4, 2013 2:12 PM
Subject: HDFS edit log NPE
 
I recently upgraded from 1.0.4 to 1.1.2.  Now however my HDFS won't start up.  There appears to be something wrong in the edits file.

Obviously I can roll back to a previous checkpoint, however it appears checkpointing has been failing for some time and my last check point is over a month old.

Is there a way to manually edit/inspect the edits file in 1.1.2 so I can fix this?  What is causing this bug?

-------------------------------------------

2013-06-04 01:07:15,952 INFO org.apache.hadoop.hdfs.server.common.Storage: Number of files = 1111
2013-06-04 01:07:16,071 INFO org.apache.hadoop.hdfs.server.common.Storage: Number of files under construction = 7
2013-06-04 01:07:16,073 INFO org.apache.hadoop.hdfs.server.common.Storage: Image file of size 270269 loaded in 0 seconds.
2013-06-04 01:07:16,075 ERROR org.apache.hadoop.hdfs.server.common.Storage: Error replaying edit log at offset 132
Recent opcode offsets: 5 14
java.lang.NullPointerException
        at org.apache.hadoop.hdfs.server.namenode.FSDirectory.addChild(FSDirectory.java:1124)
        at org.apache.hadoop.hdfs.server.namenode.FSDirectory.addChild(FSDirectory.java:1136)
        at org.apache.hadoop.hdfs.server.namenode.FSDirectory.unprotectedMkdir(FSDirectory.java:1021)
        at org.apache.hadoop.hdfs.server.namenode.FSDirectory.unprotectedMkdir(FSDirectory.java:1008)
        at org.apache.hadoop.hdfs.server.namenode.FSEditLog.loadFSEdits(FSEditLog.java:756)
        at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSEdits(FSImage.java:1025)
        at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:841)
        at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:377)
        at org.apache.hadoop.hdfs.server.namenode.FSDirectory.loadFSImage(FSDirectory.java:100)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initialize(FSNamesystem.java:411)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.java:379)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:284)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:536)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1410)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1419)
2013-06-04 01:07:16,077 ERROR org.apache.hadoop.hdfs.server.namenode.FSNamesystem: FSNamesystem initialization failed.
java.io.IOException: Error replaying edit log at offset 132
Recent opcode offsets: 5 14
        at org.apache.hadoop.hdfs.server.namenode.MetaRecoveryContext.editLogLoaderPrompt(MetaRecoveryContext.java:84)
        at org.apache.hadoop.hdfs.server.namenode.FSEditLog.loadFSEdits(FSEditLog.java:929)
        at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSEdits(FSImage.java:1025)
        at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:841)
        at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:377)
        at org.apache.hadoop.hdfs.server.namenode.FSDirectory.loadFSImage(FSDirectory.java:100)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initialize(FSNamesystem.java:411)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.java:379)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:284)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:536)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1410)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1419)
2013-06-04 01:07:16,078 ERROR org.apache.hadoop.hdfs.server.namenode.NameNode: java.io.IOException: Error replaying edit log at offset 132
Recent opcode offsets: 5 14
        at org.apache.hadoop.hdfs.server.namenode.MetaRecoveryContext.editLogLoaderPrompt(MetaRecoveryContext.java:84)
        at org.apache.hadoop.hdfs.server.namenode.FSEditLog.loadFSEdits(FSEditLog.java:929)
        at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSEdits(FSImage.java:1025)
        at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:841)
        at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:377)
        at org.apache.hadoop.hdfs.server.namenode.FSDirectory.loadFSImage(FSDirectory.java:100)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initialize(FSNamesystem.java:411)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.java:379)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:284)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:536)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1410)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1419)

2013-06-04 01:07:16,078 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG: