Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase, mail # user - Region server crash while syncing HLog


Copy link to this message
-
Re: Region server crash while syncing HLog
Ted Yu 2013-01-23, 18:04
Side note: 0.94.3 is not certified for hadoop 2.0

Cheers

On Wed, Jan 23, 2013 at 10:01 AM, ramkrishna vasudevan <
[EMAIL PROTECTED]> wrote:

> This is something to do with HDFS also.  Regarding data loss, the client
> should have known that the put has failed.  So there should not be silent
> data loss as far as i can think of.
> Any way further investigation can make things clearer.
>
> Regards
> Ram
>
> On Wed, Jan 23, 2013 at 11:29 PM, Kevin O'dell <[EMAIL PROTECTED]
> >wrote:
>
> > Varun,
> >
> >   I suspect there might be a FATAL or ERROR above that message?  That
> > message typically means the HLog can't write to HDFS, you may want to
> look
> > at the DN logs around that time.
> >
> > On Wed, Jan 23, 2013 at 12:57 PM, Varun Sharma <[EMAIL PROTECTED]>
> > wrote:
> >
> > > Hi,
> > >
> > > I had 2 region servers crash with this (running hbase-0.94.3 with
> > > hadoop-2.0.0). There is also stuff saying that Memstore was rolled
> back.
> > Is
> > > this some kind of data loss happening ?
> > >
> > > 2013-01-23 09:14:23,119 FATAL
> > > org.apache.hadoop.hbase.regionserver.wal.HLog: Could not sync.
> Requesting
> > > close of hlog
> > > java.io.IOException: Reflection
> > >         at
> > >
> > >
> >
> org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter.sync(SequenceFileLogWriter.java:304)
> > >         at
> > > org.apache.hadoop.hbase.regionserver.wal.HLog.syncer(HLog.java:1328)
> > >         at
> > > org.apache.hadoop.hbase.regionserver.wal.HLog.sync(HLog.java:1437)
> > >         at
> > >
> >
> org.apache.hadoop.hbase.regionserver.HRegion.syncOrDefer(HRegion.java:5205)
> > >         at
> > >
> > >
> >
> org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:2245)
> > >         at
> > >
> >
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:1973)
> > >         at
> > >
> > >
> >
> org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3493)
> > >         at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source)
> > >         at
> > >
> > >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > >         at java.lang.reflect.Method.invoke(Method.java:597)
> > >         at
> > >
> > >
> >
> org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
> > >         at
> > >
> >
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1438)
> > > Caused by: java.lang.reflect.InvocationTargetException
> > >         at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
> > >         at
> > >
> > >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > >         at java.lang.reflect.Method.invoke(Method.java:597)
> > >         at
> > >
> > >
> >
> org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter.sync(SequenceFileLogWriter.java:302)
> > >         ... 11 more
> > > Caused by:
> > >
> > >
> >
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException):
> > > No lease on
> > >
> > >
> >
> /hbase/.logs/ip-10-10-162-152.ec2.internal,60020,1358833785984/ip-10-10-162-152.ec2.internal%2C60020%2C1358833785984.1358932305725
> > > File does not exist. Holder DFSClient_NONMAPREDUCE_-1821696410_41 does
> > not
> > > have any open files.
> > >         at
> > >
> > >
> >
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:2308)
> > >         at
> > >
> > >
> >
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:2299)
> > >         at
> > >
> > >
> >
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalDatanode(FSNamesystem.java:2231)
> > >         at
> > >
> > >
> >
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getAdditionalDatanode(NameNodeRpcServer.java:501)
> > >         at
> > >
> > >
> >
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getAdditionalDatanode(ClientNamenodeProtocolServerSideTranslatorPB.java:316)