Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce >> mail # user >> NN stopped and cannot recover with error "There appears to be a gap in the edit log"


Copy link to this message
-
NN stopped and cannot recover with error "There appears to be a gap in the edit log"
Hi there,

 

I deployed a single node for testing, today the NN stopped and cannot start
it with eror: There appears to be a gap in the edit log.

 

2013-11-14 15:00:01,431 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
shutdown complete.
2013-11-14 15:00:01,432 FATAL
org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode join
java.io.IOException: There appears to be a gap in the edit log.  We expected
txid 8364, but got txid 27381.
       at
org.apache.hadoop.hdfs.server.namenode.MetaRecoveryContext.editLogLoaderProm
pt(MetaRecoveryContext.java:94)
       at
org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEdi
tLogLoader.java:158)
       at
org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLog
Loader.java:92)
       at
org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:744)
       at
org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:660)
       at
org.apache.hadoop.hdfs.server.namenode.FSImage.doUpgrade(FSImage.java:349)
       at
org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage
.java:261)
       at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem
.java:639)
       at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesyste
m.java:476)
       at
org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java
:403)
       at
org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:437
)
       at
org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:613)
       at
org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:598)
       at
org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java
:1169)
       at
org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1233)
2013-11-14 15:00:01,445 INFO org.apache.hadoop.util.ExitUtil: Exiting with
status 1
2013-11-14 15:00:01,448 INFO
org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at ubcdh/10.0.0.4
************************************************************/

 

Since there is only one node so restore editlogs is not available, and
hadoop namenode -recover also not fit for this situation.

 

How can I fix this issue?

 

 

JOSHUA TU JUNXIONG

Best regards

 

NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB