Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
MapReduce >> mail # user >> issue about NN autofailover happened each 24 hours


Copy link to this message
-
issue about NN autofailover happened each 24 hours
hi,all:
     i have a 5node hadoop cluster ,and i deploy  HA autofailover . but i
find the failover happened each 24 hours,i do not know why
here is my NN log , zkfc log ,zookeeper log content:

*NN log:*
2013-11-03 23:28:36,740 INFO
org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at
147925
2013-11-03 23:30:10,103 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Stopping services
started for active state
2013-11-03 23:30:10,103 INFO
org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 147925
2013-11-03 23:30:10,104 INFO
org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
Total time for transactions(ms): 1Number of transactions batched in Sync
s: 0 Number of syncs: 1 SyncTimes(ms): 11 52 55
2013-11-03 23:30:10,170 INFO
org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2
Total time for transactions(ms): 1Number of transactions batched in Sync
s: 0 Number of syncs: 2 SyncTimes(ms): 33 74 77
2013-11-03 23:30:10,222 INFO
org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Finalizing edits
file /data/namespace/1/current/edits_inprogress_0000000000000147925 -> /
data/namespace/1/current/edits_0000000000000147925-0000000000000147926
2013-11-03 23:30:10,222 INFO
org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Finalizing edits
file /data/namespace/2/current/edits_inprogress_0000000000000147925 -> /
data/namespace/2/current/edits_0000000000000147925-0000000000000147926
2013-11-03 23:30:10,222 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Starting services
required for standby state
2013-11-03 23:30:10,224 INFO
org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer: Will roll logs on
active node at ch12/192.168.11.12:8020 every 120 seconds.
2013-11-03 23:30:10,224 INFO
org.apache.hadoop.hdfs.server.namenode.ha.StandbyCheckpointer: Starting
standby checkpoint thread...
Checkpointing active NN at ch12:50070
Serving checkpoints at ch11/192.168.11.11:50070
2013-11-03 23:30:10,225 ERROR
org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException
as:hbase (auth:SIMPLE) cause:org.apache.hadoop.ipc.StandbyException: Op
eration category WRITE is not supported in state standby
2013-11-03 23:30:10,225 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 8 on 8020, call
org.apache.hadoop.hdfs.protocol.ClientProtocol.renewLease from
192.168.11.13:54864:
 error: org.apache.hadoop.ipc.StandbyException: Operation category WRITE is
not supported in state standby
2013-11-03 23:30:11,524 ERROR
org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException
as:hbase (auth:SIMPLE) cause:org.apache.hadoop.ipc.StandbyException: Op
eration category WRITE is not supported in state standby
2013-11-03 23:30:11,524 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 5 on 8020, call
org.apache.hadoop.hdfs.protocol.ClientProtocol.renewLease from
192.168.11.13:54864:
 error: org.apache.hadoop.ipc.StandbyException: Operation category WRITE is
not supported in state standby
2013-11-03 23:30:15,956 ERROR
org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException
as:hbase (auth:SIMPLE) cause:org.apache.hadoop.ipc.StandbyException: Op
eration category WRITE is not supported in state standby
2013-11-03 23:30:15,956 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 0 on 8020, call
org.apache.hadoop.hdfs.protocol.ClientProtocol.renewLease from
192.168.11.15:56050: error: org.apache.hadoop.ipc.StandbyException:
Operation category WRITE is not supported in state standby
2013-11-03 23:30:19,902 ERROR
org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException
as:hbase (auth:SIMPLE) cause:org.apache.hadoop.ipc.StandbyException:
Operation category WRITE is not supported in state standby
2013-11-03 23:30:19,902 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 2 on 8020, call
org.apache.hadoop.hdfs.protocol.ClientProtocol.renewLease from
192.168.11.12:45839: error: org.apache.hadoop.ipc.StandbyException:
Operation category WRITE is not supported in state standby
2013-11-03 23:30:21,386 ERROR
org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException
as:hbase (auth:SIMPLE) cause:org.apache.hadoop.ipc.StandbyException:
Operation category READ is not supported in state standby
2013-11-03 23:30:21,387 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 4 on 8020, call
org.apache.hadoop.hdfs.protocol.ClientProtocol.getListing from
192.168.11.11:49854: error: org.apache.hadoop.ipc.StandbyException:
Operation category READ is not supported in state standby
2013-11-03 23:30:25,215 ERROR
org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException
as:hbase (auth:SIMPLE) cause:org.apache.hadoop.ipc.StandbyException:
Operation category WRITE is not supported in state standby
2013-11-03 23:30:25,215 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 5 on 8020, call
org.apache.hadoop.hdfs.protocol.ClientProtocol.renewLease from
192.168.11.14:40328: error: org.apache.hadoop.ipc.StandbyException:
Operation category WRITE is not supported in state standby
2013-11-03 23:30:49,513 ERROR
org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException
as:mapred (auth:SIMPLE) cause:org.apache.hadoop.ipc.StandbyException:
Operation category READ is not supported in state standby
2013-11-03 23:30:49,513 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 1 on 8020, call
org.apache.hadoop.hdfs.protocol.ClientProtocol.getListing from
192.168.11.18:55750: error: org.apache.hadoop.ipc.StandbyException:
Operation category READ is not supported in state standby
2013-11-03 23:32:10,326 INFO
org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer: Triggering log
roll on remote NameNode ch12/192.168.11.12:8020
2013-11-03 23:32:10,741 INFO
org.apache.hadoop.hdfs.server.namenode.FSImage: Reading
org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream@172efb72expecting
start txid #147927
2013-11-03 23:32:10,741 INFO
org
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB