Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
HBase >> mail # user >> Region server crash while syncing HLog


Copy link to this message
-
Region server crash while syncing HLog
Hi,

I had 2 region servers crash with this (running hbase-0.94.3 with
hadoop-2.0.0). There is also stuff saying that Memstore was rolled back. Is
this some kind of data loss happening ?

2013-01-23 09:14:23,119 FATAL
org.apache.hadoop.hbase.regionserver.wal.HLog: Could not sync. Requesting
close of hlog
java.io.IOException: Reflection
        at
org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter.sync(SequenceFileLogWriter.java:304)
        at
org.apache.hadoop.hbase.regionserver.wal.HLog.syncer(HLog.java:1328)
        at
org.apache.hadoop.hbase.regionserver.wal.HLog.sync(HLog.java:1437)
        at
org.apache.hadoop.hbase.regionserver.HRegion.syncOrDefer(HRegion.java:5205)
        at
org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:2245)
        at
org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:1973)
        at
org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3493)
        at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source)
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at
org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
        at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1438)
Caused by: java.lang.reflect.InvocationTargetException
        at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at
org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter.sync(SequenceFileLogWriter.java:302)
        ... 11 more
Caused by:
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException):
No lease on
/hbase/.logs/ip-10-10-162-152.ec2.internal,60020,1358833785984/ip-10-10-162-152.ec2.internal%2C60020%2C1358833785984.1358932305725
File does not exist. Holder DFSClient_NONMAPREDUCE_-1821696410_41 does not
have any open files.
        at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:2308)
        at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:2299)
        at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalDatanode(FSNamesystem.java:2231)
        at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getAdditionalDatanode(NameNodeRpcServer.java:501)
        at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getAdditionalDatanode(ClientNamenodeProtocolServerSideTranslatorPB.java:316)
        at
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java:44082)
        at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:453)
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:898)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1693)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1689)

                                                 4163,64-71     1%

Thanks
Varun
+
Kevin Odell 2013-01-23, 17:59
+
ramkrishna vasudevan 2013-01-23, 18:01
+
Ted Yu 2013-01-23, 18:04