Hi,

I am running on a cluster where some of the machines are loaded for other
purposes. Occasionally an HBase scan fails with the message below, and I
suspect this is caused by on or more of the region servers being overloaded
with other processes (not hadoop/hbase) and not being able to respond
correctly. Is this possible?

Thanks,

Jon

Exception in thread "main" java.lang.RuntimeException:
org.apache.hadoop.hbase.regionserver.LeaseException:
org.apache.hadoop.hbase.regionserver.LeaseException: lease
'-3573171992963675348' does not exist
    at
org.apache.hadoop.hbase.regionserver.Leases.removeLease(Leases.java:231)
    at
org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:2117)
    at sun.reflect.GeneratedMethodAccessor32.invoke(Unknown Source)
    at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:616)
    at
org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
    at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326)

    at
org.apache.hadoop.hbase.client.HTable$ClientScanner$1.hasNext(HTable.java:1402)
    at prd.Levelize.readPins(Levelize.java:50)
    at prd.Levelize.main(Levelize.java:245)
Caused by: org.apache.hadoop.hbase.regionserver.LeaseException:
org.apache.hadoop.hbase.regionserver.LeaseException: lease
'-3573171992963675348' does not exist
    at
org.apache.hadoop.hbase.regionserver.Leases.removeLease(Leases.java:231)
    at
org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:2117)
    at sun.reflect.GeneratedMethodAccessor32.invoke(Unknown Source)
    at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:616)
    at
org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
    at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326)

    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
    at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:532)
    at
org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:96)
    at
org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:84)
    at
org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:39)
    at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionServerWithRetries(HConnectionManager.java:1325)
    at
org.apache.hadoop.hbase.client.HTable$ClientScanner.next(HTable.java:1293)
    at
org.apache.hadoop.hbase.client.HTable$ClientScanner$1.hasNext(HTable.java:1399)
    ... 2 more

This exception means the scanner is expired in the region server side.
You can adjust the scanner expiration setting, or make your client fast.

Thanks,
Jimmy
On Fri, Jul 20, 2012 at 9:27 AM, Jonathan Bishop <[EMAIL PROTECTED]> wrote:
> Hi,
>
> I am running on a cluster where some of the machines are loaded for other
> purposes. Occasionally an HBase scan fails with the message below, and I
> suspect this is caused by on or more of the region servers being overloaded
> with other processes (not hadoop/hbase) and not being able to respond
> correctly. Is this possible?
>
> Thanks,
>
> Jon
>
> Exception in thread "main" java.lang.RuntimeException:
> org.apache.hadoop.hbase.regionserver.LeaseException:
> org.apache.hadoop.hbase.regionserver.LeaseException: lease
> '-3573171992963675348' does not exist
>     at
> org.apache.hadoop.hbase.regionserver.Leases.removeLease(Leases.java:231)
>     at
> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:2117)
>     at sun.reflect.GeneratedMethodAccessor32.invoke(Unknown Source)
>     at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>     at java.lang.reflect.Method.invoke(Method.java:616)
>     at
> org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
>     at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326)
>
>     at
> org.apache.hadoop.hbase.client.HTable$ClientScanner$1.hasNext(HTable.java:1402)
>     at prd.Levelize.readPins(Levelize.java:50)
>     at prd.Levelize.main(Levelize.java:245)
> Caused by: org.apache.hadoop.hbase.regionserver.LeaseException:
> org.apache.hadoop.hbase.regionserver.LeaseException: lease
> '-3573171992963675348' does not exist
>     at
> org.apache.hadoop.hbase.regionserver.Leases.removeLease(Leases.java:231)
>     at
> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:2117)
>     at sun.reflect.GeneratedMethodAccessor32.invoke(Unknown Source)
>     at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>     at java.lang.reflect.Method.invoke(Method.java:616)
>     at
> org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
>     at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326)
>
>     at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>     at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
>     at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>     at java.lang.reflect.Constructor.newInstance(Constructor.java:532)
>     at
> org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:96)
>     at
> org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:84)
>     at
> org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:39)
>     at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionServerWithRetries(HConnectionManager.java:1325)
>     at
> org.apache.hadoop.hbase.client.HTable$ClientScanner.next(HTable.java:1293)
>     at
> org.apache.hadoop.hbase.client.HTable$ClientScanner$1.hasNext(HTable.java:1399)
>     ... 2 more