Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce >> mail # user >> NN stopped and cannot recover with error "There appears to be a gap in the edit log"


Copy link to this message
-
NN stopped and cannot recover with error "There appears to be a gap in the edit log"
Hi there,

 

I deployed a single node for testing, today the NN stopped and cannot start
it with eror: There appears to be a gap in the edit log.

 

2013-11-14 15:00:01,431 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
shutdown complete.
2013-11-14 15:00:01,432 FATAL
org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode join
java.io.IOException: There appears to be a gap in the edit log.  We expected
txid 8364, but got txid 27381.
       at
org.apache.hadoop.hdfs.server.namenode.MetaRecoveryContext.editLogLoaderProm
pt(MetaRecoveryContext.java:94)
       at
org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEdi
tLogLoader.java:158)
       at
org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLog
Loader.java:92)
       at
org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:744)
       at
org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:660)
       at
org.apache.hadoop.hdfs.server.namenode.FSImage.doUpgrade(FSImage.java:349)
       at
org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage
.java:261)
       at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem
.java:639)
       at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesyste
m.java:476)
       at
org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java
:403)
       at
org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:437
)
       at
org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:613)
       at
org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:598)
       at
org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java
:1169)
       at
org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1233)
2013-11-14 15:00:01,445 INFO org.apache.hadoop.util.ExitUtil: Exiting with
status 1
2013-11-14 15:00:01,448 INFO
org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at ubcdh/10.0.0.4
************************************************************/

 

Since there is only one node so restore editlogs is not available, and
hadoop namenode -recover also not fit for this situation.

 

How can I fix this issue?

 

 

JOSHUA TU JUNXIONG

Best regards