Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
MapReduce, mail # user - issue about NN autofailover happened each 24 hours


Copy link to this message
-
issue about NN autofailover happened each 24 hours
ch huang 2013-11-04, 01:21
hi,all:
     i have a 5node hadoop cluster ,and i deploy  HA autofailover . but i
find the failover happened each 24 hours,i do not know why
here is my NN log , zkfc log ,zookeeper log content:

*NN log:*
2013-11-03 23:28:36,740 INFO
org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
147925
2013-11-03 23:30:10,103 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Stopping services
started for active state
2013-11-03 23:30:10,103 INFO
org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 147925
2013-11-03 23:30:10,104 INFO
org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
Total time for transactions(ms): 1Number of transactions batched in Sync
s: 0 Number of syncs: 1 SyncTimes(ms): 11 52 55
2013-11-03 23:30:10,170 INFO
org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
Total time for transactions(ms): 1Number of transactions batched in Sync
s: 0 Number of syncs: 2 SyncTimes(ms): 33 74 77
2013-11-03 23:30:10,222 INFO
org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Finalizing edits
file /data/namespace/1/current/edits_inprogress_0000000000000147925 -> /
data/namespace/1/current/edits_0000000000000147925-0000000000000147926
2013-11-03 23:30:10,222 INFO
org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Finalizing edits
file /data/namespace/2/current/edits_inprogress_0000000000000147925 -> /
data/namespace/2/current/edits_0000000000000147925-0000000000000147926
2013-11-03 23:30:10,222 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Starting services
required for standby state
2013-11-03 23:30:10,224 INFO
org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer: Will roll logs on
active node at ch12/192.168.11.12:8020 every 120 seconds.
2013-11-03 23:30:10,224 INFO
org.apache.hadoop.hdfs.server.namenode.ha.StandbyCheckpointer: Starting
standby checkpoint thread...
Checkpointing active NN at ch12:50070
Serving checkpoints at ch11/192.168.11.11:50070
2013-11-03 23:30:10,225 ERROR
org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException
as:hbase (auth:SIMPLE) cause:org.apache.hadoop.ipc.StandbyException: Op
eration category WRITE is not supported in state standby
2013-11-03 23:30:10,225 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 8 on 8020, call
org.apache.hadoop.hdfs.protocol.ClientProtocol.renewLease from
192.168.11.13:54864:
 error: org.apache.hadoop.ipc.StandbyException: Operation category WRITE is
not supported in state standby
2013-11-03 23:30:11,524 ERROR
org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException
as:hbase (auth:SIMPLE) cause:org.apache.hadoop.ipc.StandbyException: Op
eration category WRITE is not supported in state standby
2013-11-03 23:30:11,524 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 5 on 8020, call
org.apache.hadoop.hdfs.protocol.ClientProtocol.renewLease from
192.168.11.13:54864:
 error: org.apache.hadoop.ipc.StandbyException: Operation category WRITE is
not supported in state standby
2013-11-03 23:30:15,956 ERROR
org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException
as:hbase (auth:SIMPLE) cause:org.apache.hadoop.ipc.StandbyException: Op
eration category WRITE is not supported in state standby
2013-11-03 23:30:15,956 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 0 on 8020, call
org.apache.hadoop.hdfs.protocol.ClientProtocol.renewLease from
192.168.11.15:56050: error: org.apache.hadoop.ipc.StandbyException:
Operation category WRITE is not supported in state standby
2013-11-03 23:30:19,902 ERROR
org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException
as:hbase (auth:SIMPLE) cause:org.apache.hadoop.ipc.StandbyException:
Operation category WRITE is not supported in state standby
2013-11-03 23:30:19,902 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 2 on 8020, call
org.apache.hadoop.hdfs.protocol.ClientProtocol.renewLease from
192.168.11.12:45839: error: org.apache.hadoop.ipc.StandbyException:
Operation category WRITE is not supported in state standby
2013-11-03 23:30:21,386 ERROR
org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException
as:hbase (auth:SIMPLE) cause:org.apache.hadoop.ipc.StandbyException:
Operation category READ is not supported in state standby
2013-11-03 23:30:21,387 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 4 on 8020, call
org.apache.hadoop.hdfs.protocol.ClientProtocol.getListing from
192.168.11.11:49854: error: org.apache.hadoop.ipc.StandbyException:
Operation category READ is not supported in state standby
2013-11-03 23:30:25,215 ERROR
org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException
as:hbase (auth:SIMPLE) cause:org.apache.hadoop.ipc.StandbyException:
Operation category WRITE is not supported in state standby
2013-11-03 23:30:25,215 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 5 on 8020, call
org.apache.hadoop.hdfs.protocol.ClientProtocol.renewLease from
192.168.11.14:40328: error: org.apache.hadoop.ipc.StandbyException:
Operation category WRITE is not supported in state standby
2013-11-03 23:30:49,513 ERROR
org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException
as:mapred (auth:SIMPLE) cause:org.apache.hadoop.ipc.StandbyException:
Operation category READ is not supported in state standby
2013-11-03 23:30:49,513 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 1 on 8020, call
org.apache.hadoop.hdfs.protocol.ClientProtocol.getListing from
192.168.11.18:55750: error: org.apache.hadoop.ipc.StandbyException:
Operation category READ is not supported in state standby
2013-11-03 23:32:10,326 INFO
org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer: Triggering log
roll on remote NameNode ch12/192.168.11.12:8020
2013-11-03 23:32:10,741 INFO
org.apache.hadoop.hdfs.server.namenode.FSImage: Reading
org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream@172efb72expecting
start txid #147927
2013-11-03 23:32:10,741 INFO
org