Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
HBase >> mail # user >> Region server crash while syncing HLog


+
Varun Sharma 2013-01-23, 17:57
+
Kevin Odell 2013-01-23, 17:59
+
ramkrishna vasudevan 2013-01-23, 18:01
Copy link to this message
-
Re: Region server crash while syncing HLog
Side note: 0.94.3 is not certified for hadoop 2.0

Cheers

On Wed, Jan 23, 2013 at 10:01 AM, ramkrishna vasudevan <
[EMAIL PROTECTED]> wrote:

> This is something to do with HDFS also.  Regarding data loss, the client
> should have known that the put has failed.  So there should not be silent
> data loss as far as i can think of.
> Any way further investigation can make things clearer.
>
> Regards
> Ram
>
> On Wed, Jan 23, 2013 at 11:29 PM, Kevin O'dell <[EMAIL PROTECTED]
> >wrote:
>
> > Varun,
> >
> >   I suspect there might be a FATAL or ERROR above that message?  That
> > message typically means the HLog can't write to HDFS, you may want to
> look
> > at the DN logs around that time.
> >
> > On Wed, Jan 23, 2013 at 12:57 PM, Varun Sharma <[EMAIL PROTECTED]>
> > wrote:
> >
> > > Hi,
> > >
> > > I had 2 region servers crash with this (running hbase-0.94.3 with
> > > hadoop-2.0.0). There is also stuff saying that Memstore was rolled
> back.
> > Is
> > > this some kind of data loss happening ?
> > >
> > > 2013-01-23 09:14:23,119 FATAL
> > > org.apache.hadoop.hbase.regionserver.wal.HLog: Could not sync.
> Requesting
> > > close of hlog
> > > java.io.IOException: Reflection
> > >         at
> > >
> > >
> >
> org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter.sync(SequenceFileLogWriter.java:304)
> > >         at
> > > org.apache.hadoop.hbase.regionserver.wal.HLog.syncer(HLog.java:1328)
> > >         at
> > > org.apache.hadoop.hbase.regionserver.wal.HLog.sync(HLog.java:1437)
> > >         at
> > >
> >
> org.apache.hadoop.hbase.regionserver.HRegion.syncOrDefer(HRegion.java:5205)
> > >         at
> > >
> > >
> >
> org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:2245)
> > >         at
> > >
> >
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:1973)
> > >         at
> > >
> > >
> >
> org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3493)
> > >         at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source)
> > >         at
> > >
> > >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > >         at java.lang.reflect.Method.invoke(Method.java:597)
> > >         at
> > >
> > >
> >
> org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
> > >         at
> > >
> >
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1438)
> > > Caused by: java.lang.reflect.InvocationTargetException
> > >         at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
> > >         at
> > >
> > >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > >         at java.lang.reflect.Method.invoke(Method.java:597)
> > >         at
> > >
> > >
> >
> org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter.sync(SequenceFileLogWriter.java:302)
> > >         ... 11 more
> > > Caused by:
> > >
> > >
> >
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException):
> > > No lease on
> > >
> > >
> >
> /hbase/.logs/ip-10-10-162-152.ec2.internal,60020,1358833785984/ip-10-10-162-152.ec2.internal%2C60020%2C1358833785984.1358932305725
> > > File does not exist. Holder DFSClient_NONMAPREDUCE_-1821696410_41 does
> > not
> > > have any open files.
> > >         at
> > >
> > >
> >
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:2308)
> > >         at
> > >
> > >
> >
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:2299)
> > >         at
> > >
> > >
> >
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalDatanode(FSNamesystem.java:2231)
> > >         at
> > >
> > >
> >
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getAdditionalDatanode(NameNodeRpcServer.java:501)
> > >         at
> > >
> > >
> >
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getAdditionalDatanode(ClientNamenodeProtocolServerSideTranslatorPB.java:316)
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB