Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
HBase, mail # user - Differences between ScannerTimeoutException and LeaseException during a scanner


+
Guillaume Perrot 2012-11-15, 14:46
+
Guillaume Perrot 2012-11-15, 15:45
+
Ted Yu 2012-11-15, 17:43
Copy link to this message
-
Re: Differences between ScannerTimeoutException and LeaseException during a scanner
Guillaume Perrot 2012-11-15, 18:12
No, but we'll try that soon.

2012/11/15 Ted Yu <[EMAIL PROTECTED]>

> Thanks for sharing your experience.
>
> Have you set hbase.rpc.timeout > hbase.regionserver.lease.period in your
> environment ?
>
> Cheers
>
> On Thu, Nov 15, 2012 at 7:45 AM, Guillaume Perrot <[EMAIL PROTECTED]
> >wrote:
>
> > It looks like a "race condition" between the check that the lease is
> > expired in HRegionServer.next, and the LeaseListener.leaseExpired
> methods.
> > If the next call from the client is done before the lease listener do its
> > job, the scanner still exists in the map but is expired, which causes
> > LeaseException instead of UnknownScannerException.
> > Indeed I found this in the logs:
> >
> > 2012-11-15 08:45:34,003 ERROR
> > org.apache.hadoop.hbase.regionserver.HRegionServer:
> > org.apache.hadoop.hbase.regionserver.LeaseException: lease
> > '4939065839430667309' does not exist
> >  at
> > org.apache.hadoop.hbase.regionserver.Leases.removeLease(Leases.java:231)
> > at
> >
> >
> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:2117)
> >  at sun.reflect.GeneratedMethodAccessor31.invoke(Unknown Source)
> > at
> >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >  at java.lang.reflect.Method.invoke(Method.java:597)
> > at
> >
> >
> org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
> >  at
> >
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326)
> >
> > Then
> >
> > 2012-11-15 08:48:37,734 INFO
> > org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner
> > 4939065839430667309 lease expired
> >
> > The scanner is properly closed way too late!
> > I just realized there is already a related issue about this:
> > https://issues.apache.org/jira/browse/HBASE-6856
> >
> > 2012/11/15 Guillaume Perrot <[EMAIL PROTECTED]>
> >
> > > Hi,
> > > Similar questions have been mailed in the past, but they are not clear
> > > enough to me, so here is a new one:
> > > In HBase 0.90.3, when a client scanner was too slow to advance (no
> actual
> > > next call within 60s), we had a nice ScannerTimeoutException in client
> > > (UnknownScannerException in region server).
> > > With HBase 0.92.1, when I simulate such a slow scanner I still have
> this
> > > exception.
> > >
> > > But in our production environment, scanners that used to
> > > produce ScannerTimeoutException in HBase 0.90.3 no longer produce them
> > > since we upgraded to HBase 0.92.1.
> > > Instead we now have:
> > > org.apache.hadoop.hbase.regionserver.LeaseException:
> > > org.apache.hadoop.hbase.regionserver.LeaseException: lease
> > > '2770824890045864909' does not exist
> > > at
> > org.apache.hadoop.hbase.regionserver.Leases.removeLease(Leases.java:231)
> > >  at
> > >
> >
> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:2117)
> > > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source)
> > >  at
> > >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > > at java.lang.reflect.Method.invoke(Method.java:597)
> > >  at
> > >
> >
> org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
> > > at
> > >
> >
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326)
> > >
> > > I don't uderstand the workflow leading to this exception instead of a
> > > ScannerTimeoutException, any local attempt to reproduce this problem
> > leads
> > > to a ScannerTimeoutException and not a LeaseException.
> > > Any idea ?
> > > Thanks in advance.
> > >
> >
> >
> >
> > --
> >   *Guillaume Perrot
> > **Software Engineer
> > *[image: logo]
> >    *Contact info *
> > [EMAIL PROTECTED]
> > www.capptain.com
> >
> >  *Rennes Office*
> > Office: +33 2 99 65 69 13
> > 10 rue Jean-Marie Duhamel
> > 35000 Rennes
> > France *Paris Office*
> > Office: +33 1 84 06 13 85
> > Fax: +33 9 57 72 20 18
> > 18 rue Tronchet
> > 75008 Paris
> > France
> >
>
+
killaron 2013-02-12, 23:26