Hi,
Similar questions have been mailed in the past, but they are not clear
enough to me, so here is a new one:
In HBase 0.90.3, when a client scanner was too slow to advance (no actual
next call within 60s), we had a nice ScannerTimeoutException in client
(UnknownScannerException in region server).
With HBase 0.92.1, when I simulate such a slow scanner I still have this
exception.

But in our production environment, scanners that used to
produce ScannerTimeoutException in HBase 0.90.3 no longer produce them
since we upgraded to HBase 0.92.1.
Instead we now have:
org.apache.hadoop.hbase.regionserver.LeaseException:
org.apache.hadoop.hbase.regionserver.LeaseException: lease
'2770824890045864909' does not exist
at org.apache.hadoop.hbase.regionserver.Leases.removeLease(Leases.java:231)
 at
org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:2117)
at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source)
 at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
 at
org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326)

I don't uderstand the workflow leading to this exception instead of a
ScannerTimeoutException, any local attempt to reproduce this problem leads
to a ScannerTimeoutException and not a LeaseException.
Any idea ?
Thanks in advance.

It looks like a "race condition" between the check that the lease is
expired in HRegionServer.next, and the LeaseListener.leaseExpired methods.
If the next call from the client is done before the lease listener do its
job, the scanner still exists in the map but is expired, which causes
LeaseException instead of UnknownScannerException.
Indeed I found this in the logs:

2012-11-15 08:45:34,003 ERROR
org.apache.hadoop.hbase.regionserver.HRegionServer:
org.apache.hadoop.hbase.regionserver.LeaseException: lease
'4939065839430667309' does not exist
 at org.apache.hadoop.hbase.regionserver.Leases.removeLease(Leases.java:231)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:2117)
 at sun.reflect.GeneratedMethodAccessor31.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
 at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
 at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326)

Then

2012-11-15 08:48:37,734 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner
4939065839430667309 lease expired

The scanner is properly closed way too late!
I just realized there is already a related issue about this:
https://issues.apache.org/jira/browse/HBASE-6856

2012/11/15 Guillaume Perrot <[EMAIL PROTECTED]>

> Hi,
> Similar questions have been mailed in the past, but they are not clear
> enough to me, so here is a new one:
> In HBase 0.90.3, when a client scanner was too slow to advance (no actual
> next call within 60s), we had a nice ScannerTimeoutException in client
> (UnknownScannerException in region server).
> With HBase 0.92.1, when I simulate such a slow scanner I still have this
> exception.
>
> But in our production environment, scanners that used to
> produce ScannerTimeoutException in HBase 0.90.3 no longer produce them
> since we upgraded to HBase 0.92.1.
> Instead we now have:
> org.apache.hadoop.hbase.regionserver.LeaseException:
> org.apache.hadoop.hbase.regionserver.LeaseException: lease
> '2770824890045864909' does not exist
> at org.apache.hadoop.hbase.regionserver.Leases.removeLease(Leases.java:231)
>  at
> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:2117)
> at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source)
>  at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
>  at
> org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
> at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326)
>
> I don't uderstand the workflow leading to this exception instead of a
> ScannerTimeoutException, any local attempt to reproduce this problem leads
> to a ScannerTimeoutException and not a LeaseException.
> Any idea ?
> Thanks in advance.
>

--
  *Guillaume Perrot
**Software Engineer
*[image: logo]
   *Contact info *
[EMAIL PROTECTED]
www.capptain.com

 *Rennes Office*
Office: +33 2 99 65 69 13
10 rue Jean-Marie Duhamel
35000 Rennes
France *Paris Office*
Office: +33 1 84 06 13 85
Fax: +33 9 57 72 20 18
18 rue Tronchet
75008 Paris
France

IMPORTANT NOTICE – UBIKOD and CAPPTAIN are registered trademarks of UBIKOD
S.A.R.L., all copyrights are reserved. The contents of this email and
attachments are confidential and may be subject to legal privilege and/or
protected by copyright. Copying or communicating any part of it to others
is prohibited and may be unlawful. If you are not the intended recipient
you must not use, copy, distribute or rely on this email and should please
return it immediately or notify us by telephone. At present the integrity
of email across the Internet cannot be guaranteed. Therefore UBIKOD
S.A.R.L. will not accept liability for any claims arising as a result of
the use of this medium for transmissions by or to UBIKOD S.A.R.L.. UBIKOD
S.A.R.L. may exercise any of its rights under relevant law, to monitor the
content of all electronic communications. You should therefore be aware
that this communication and any responses might have been monitored, and
may be accessed by UBIKOD S.A.R.L. The views expressed in this document are
that of the individual and may not necessarily constitute or imply its
endorsement or recommendation by UBIKOD S.A.R.L. The content of this
electronic mail may be subject to the confidentiality terms of a
"Non-Disclosure Agreement" (NDA).

Thanks for sharing your experience.

Have you set hbase.rpc.timeout > hbase.regionserver.lease.period in your
environment ?

Cheers

On Thu, Nov 15, 2012 at 7:45 AM, Guillaume Perrot <[EMAIL PROTECTED]>wrote:

> It looks like a "race condition" between the check that the lease is
> expired in HRegionServer.next, and the LeaseListener.leaseExpired methods.
> If the next call from the client is done before the lease listener do its
> job, the scanner still exists in the map but is expired, which causes
> LeaseException instead of UnknownScannerException.
> Indeed I found this in the logs:
>
> 2012-11-15 08:45:34,003 ERROR
> org.apache.hadoop.hbase.regionserver.HRegionServer:
> org.apache.hadoop.hbase.regionserver.LeaseException: lease
> '4939065839430667309' does not exist
>  at
> org.apache.hadoop.hbase.regionserver.Leases.removeLease(Leases.java:231)
> at
>
> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:2117)
>  at sun.reflect.GeneratedMethodAccessor31.invoke(Unknown Source)
> at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>  at java.lang.reflect.Method.invoke(Method.java:597)
> at
>
> org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
>  at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326)
>
> Then
>
> 2012-11-15 08:48:37,734 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner
> 4939065839430667309 lease expired
>
> The scanner is properly closed way too late!
> I just realized there is already a related issue about this:
> https://issues.apache.org/jira/browse/HBASE-6856
>
> 2012/11/15 Guillaume Perrot <[EMAIL PROTECTED]>
>
> > Hi,
> > Similar questions have been mailed in the past, but they are not clear
> > enough to me, so here is a new one:
> > In HBase 0.90.3, when a client scanner was too slow to advance (no actual
> > next call within 60s), we had a nice ScannerTimeoutException in client
> > (UnknownScannerException in region server).
> > With HBase 0.92.1, when I simulate such a slow scanner I still have this
> > exception.
> >
> > But in our production environment, scanners that used to
> > produce ScannerTimeoutException in HBase 0.90.3 no longer produce them
> > since we upgraded to HBase 0.92.1.
> > Instead we now have:
> > org.apache.hadoop.hbase.regionserver.LeaseException:
> > org.apache.hadoop.hbase.regionserver.LeaseException: lease
> > '2770824890045864909' does not exist
> > at
> org.apache.hadoop.hbase.regionserver.Leases.removeLease(Leases.java:231)
> >  at
> >
> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:2117)
> > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source)
> >  at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> >  at
> >
> org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
> > at
> >
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326)
> >
> > I don't uderstand the workflow leading to this exception instead of a
> > ScannerTimeoutException, any local attempt to reproduce this problem
> leads
> > to a ScannerTimeoutException and not a LeaseException.
> > Any idea ?
> > Thanks in advance.
> >
>
>
>
> --
>   *Guillaume Perrot
> **Software Engineer
> *[image: logo]
>    *Contact info *
> [EMAIL PROTECTED]
> www.capptain.com
>
>  *Rennes Office*
> Office: +33 2 99 65 69 13
> 10 rue Jean-Marie Duhamel
> 35000 Rennes
> France *Paris Office*
> Office: +33 1 84 06 13 85
> Fax: +33 9 57 72 20 18
> 18 rue Tronchet
> 75008 Paris
> France
>
> IMPORTANT NOTICE – UBIKOD and CAPPTAIN are registered trademarks of UBIKOD
> S.A.R.L., all copyrights are reserved. The contents of this email and
> attachments are confidential and may be subject to legal privilege and/or
> protected by copyright. Copying or communicating any part of it to others
> is prohibited and may be unlawful. If you are not the intended recipient

No, but we'll try that soon.

2012/11/15 Ted Yu <[EMAIL PROTECTED]>

> Thanks for sharing your experience.
>
> Have you set hbase.rpc.timeout > hbase.regionserver.lease.period in your
> environment ?
>
> Cheers
>
> On Thu, Nov 15, 2012 at 7:45 AM, Guillaume Perrot <[EMAIL PROTECTED]
> >wrote:
>
> > It looks like a "race condition" between the check that the lease is
> > expired in HRegionServer.next, and the LeaseListener.leaseExpired
> methods.
> > If the next call from the client is done before the lease listener do its
> > job, the scanner still exists in the map but is expired, which causes
> > LeaseException instead of UnknownScannerException.
> > Indeed I found this in the logs:
> >
> > 2012-11-15 08:45:34,003 ERROR
> > org.apache.hadoop.hbase.regionserver.HRegionServer:
> > org.apache.hadoop.hbase.regionserver.LeaseException: lease
> > '4939065839430667309' does not exist
> >  at
> > org.apache.hadoop.hbase.regionserver.Leases.removeLease(Leases.java:231)
> > at
> >
> >
> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:2117)
> >  at sun.reflect.GeneratedMethodAccessor31.invoke(Unknown Source)
> > at
> >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >  at java.lang.reflect.Method.invoke(Method.java:597)
> > at
> >
> >
> org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
> >  at
> >
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326)
> >
> > Then
> >
> > 2012-11-15 08:48:37,734 INFO
> > org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner
> > 4939065839430667309 lease expired
> >
> > The scanner is properly closed way too late!
> > I just realized there is already a related issue about this:
> > https://issues.apache.org/jira/browse/HBASE-6856
> >
> > 2012/11/15 Guillaume Perrot <[EMAIL PROTECTED]>
> >
> > > Hi,
> > > Similar questions have been mailed in the past, but they are not clear
> > > enough to me, so here is a new one:
> > > In HBase 0.90.3, when a client scanner was too slow to advance (no
> actual
> > > next call within 60s), we had a nice ScannerTimeoutException in client
> > > (UnknownScannerException in region server).
> > > With HBase 0.92.1, when I simulate such a slow scanner I still have
> this
> > > exception.
> > >
> > > But in our production environment, scanners that used to
> > > produce ScannerTimeoutException in HBase 0.90.3 no longer produce them
> > > since we upgraded to HBase 0.92.1.
> > > Instead we now have:
> > > org.apache.hadoop.hbase.regionserver.LeaseException:
> > > org.apache.hadoop.hbase.regionserver.LeaseException: lease
> > > '2770824890045864909' does not exist
> > > at
> > org.apache.hadoop.hbase.regionserver.Leases.removeLease(Leases.java:231)
> > >  at
> > >
> >
> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:2117)
> > > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source)
> > >  at
> > >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > > at java.lang.reflect.Method.invoke(Method.java:597)
> > >  at
> > >
> >
> org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
> > > at
> > >
> >
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326)
> > >
> > > I don't uderstand the workflow leading to this exception instead of a
> > > ScannerTimeoutException, any local attempt to reproduce this problem
> > leads
> > > to a ScannerTimeoutException and not a LeaseException.
> > > Any idea ?
> > > Thanks in advance.
> > >
> >
> >
> >
> > --
> >   *Guillaume Perrot
> > **Software Engineer
> > *[image: logo]
> >    *Contact info *
> > [EMAIL PROTECTED]
> > www.capptain.com
> >
> >  *Rennes Office*
> > Office: +33 2 99 65 69 13
> > 10 rue Jean-Marie Duhamel
> > 35000 Rennes
> > France *Paris Office*
> > Office: +33 1 84 06 13 85
> > Fax: +33 9 57 72 20 18
> > 18 rue Tronchet
> > 75008 Paris
> > France
> >
>

I have the following configuration:

  <property>
    <name>hbase.regionserver.lease.period</name>
    <value>600000</value>
  </property>

  <property>
    <name>hbase.rpc.timeout</name>
    <value>610000</value>
  </property>

  <property>
    <name>hbase.client.scanner.timeout.period</name>
    <value>600000</value>
  </property>
However, we continue to see hadoop.hbase.regionserver.LeaseException: lease
'-1299304091761958240' does not exist errors after 80 seconds and multiple task
retries and it leads to failed jobs.

Does anyone know why the lease expires after only 80 seconds even thought the
rpc timeout value is set to 10mins?