Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Hadoop >> mail # user >> Standby NameNode checkpoint exception


Copy link to this message
-
Standby NameNode checkpoint exception
I use hadoop-2.0.5, and QJM for HA.

When Standby NameNode do checkpoint,there are below exception  in Standby
NameNode:
2013-08-01 13:43:07,965 INFO
org.apache.hadoop.hdfs.server.namenode.ha.StandbyCheckpointer: Triggering
checkpoint because there have been 763426 txns since the last checkpoint, wh
ich exceeds the configured threshold 40000
2013-08-01 13:43:07,966 INFO
org.apache.hadoop.hdfs.server.namenode.FSImage: Saving image file
/home/musa.ll/hadoop2/cluster-data/name/current/fsimage.ckpt_0000000000048708235
usi
ng no compression
2013-08-01 13:43:37,405 INFO
org.apache.hadoop.hdfs.server.namenode.FSImage: Image file of size
1504089705 saved in 29 seconds.
2013-08-01 13:43:37,410 INFO
org.apache.hadoop.hdfs.server.namenode.NNStorageRetentionManager: Going to
retain 2 images with txid >= 47944809
2013-08-01 13:43:37,410 INFO
org.apache.hadoop.hdfs.server.namenode.NNStorageRetentionManager: Purging
old image FSImageFile(file=/home/musa.ll/hadoop2/cluster-data/name/current/f
simage_0000000000047222679, cpktTxId=0000000000047222679)
2013-08-01 13:43:37,723 WARN
org.apache.hadoop.hdfs.server.namenode.FSEditLog: Unable to determine input
streams from QJM to [10.232.98.61:20022, 10.232.98.62:20022, 10.232.98.63:
20022, 10.232.98.64:20022, 10.232.98.65:20022]. Skipping.
org.apache.hadoop.hdfs.qjournal.client.QuorumException: Got too many
exceptions to achieve quorum size 3/5. 4 exceptions thrown:
10.232.98.62:20022: Asked for firstTxId 46944810 which is in the middle of
file
/home/musa.ll/hadoop2/journal/mycluster/current/edits_0000000000046630461-0000000000047222679
        at
org.apache.hadoop.hdfs.server.namenode.FileJournalManager.getRemoteEditLogs(FileJournalManager.java:183)
        at
org.apache.hadoop.hdfs.qjournal.server.Journal.getEditLogManifest(Journal.java:628)
        at
org.apache.hadoop.hdfs.qjournal.server.JournalNodeRpcServer.getEditLogManifest(JournalNodeRpcServer.java:180)
        at
org.apache.hadoop.hdfs.qjournal.protocolPB.QJournalProtocolServerSideTranslatorPB.getEditLogManifest(QJournalProtocolServerSideTranslatorPB.java:203)
        at
org.apache.hadoop.hdfs.qjournal.protocol.QJournalProtocolProtos$QJournalProtocolService$2.callBlockingMethod(QJournalProtocolProtos.java:14028)
        at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:454)
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1014)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1741)
"hadoop-musa.ll-namenode-dw78.kgb.sqa.cm4.log" 350842L,
60353971C
348726,1      99%
2013-08-01 14:28:07,051 INFO
org.apache.hadoop.hdfs.server.namenode.TransferFsImage: Transfer took
26.08s at 0.00 KB/s
2013-08-01 14:28:07,051 INFO
org.apache.hadoop.hdfs.server.namenode.TransferFsImage: Uploaded image with
txid 60835762 to namenode at 10.232.98.77:20021
2013-08-01 14:29:05,203 INFO
org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer: Triggering log
roll on remote NameNode /10.232.98.77:20020
2013-08-01 14:29:06,242 INFO
org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader: replaying edit log:
137678/567332 transactions completed. (24%)
2013-08-01 14:29:07,243 INFO
org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader: replaying edit log:
275618/567332 transactions completed. (49%)
2013-08-01 14:29:08,244 INFO
org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader: replaying edit log:
407627/567332 transactions completed. (72%)
2013-08-01 14:29:09,245 INFO
org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader: replaying edit log:
545153/567332 transactions completed. (96%)
2013-08-01 14:29:20,146 INFO
org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer: Loaded 567332
edits starting from txid 60835762
2013-08-01 14:30:44,411 INFO
org.apache.hadoop.hdfs.server.namenode.FSImage: Image file of size
1950604672 saved in 37 seconds.
2013-08-01 14:30:44,416 INFO
org.apache.hadoop.hdfs.server.namenode.NNStorageRetentionManager: Going to
retain 2 images with txid >= 60835762
org.apache.hadoop.hdfs.qjournal.client.QuorumException: Got too many
exceptions to achieve quorum size 3/5. 4 exceptions thrown:
10.232.98.62:20022: Asked for firstTxId 59835763 which is in the middle of
file
/home/musa.ll/hadoop2/journal/mycluster/current/edits_0000000000059678382-0000000000060264590
        at
org.apache.hadoop.hdfs.server.namenode.FileJournalManager.getRemoteEditLogs(FileJournalManager.java:183)
        at
org.apache.hadoop.hdfs.qjournal.server.Journal.getEditLogManifest(Journal.java:628)
        at
org.apache.hadoop.hdfs.qjournal.server.JournalNodeRpcServer.getEditLogManifest(JournalNodeRpcServer.java:180)
        at
org.apache.hadoop.hdfs.qjournal.protocolPB.QJournalProtocolServerSideTranslatorPB.getEditLogManifest(QJournalProtocolServerSideTranslatorPB.java:203)
        at
org.apache.hadoop.hdfs.qjournal.protocol.QJournalProtocolProtos$QJournalProtocolService$2.callBlockingMethod(QJournalProtocolProtos.java:14028)
        at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:454)
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1014)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1741)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1737)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:396)
        at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1478)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1735)

10.232.98.63:20022: Asked for firstTxId 59835763 which is in the middle of
file
/home/musa.ll/hadoop2/journal/mycluster/current/edits_0000000000059678382-0000000000060264590
        at
org.apache.hadoop.hdfs.server.namenode.FileJournalManager.getRemoteEditLogs(FileJournalManager.java:183)
        at
org.apache.hadoop.hdfs.qjournal.server.Journal.getEditLogManifest(Journal.java:628)
        at
org.apache.hadoop.hdfs.qjournal.protocol.QJournalP
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB