Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # user >> Is this exception caused by an overloaded node?


Copy link to this message
-
Is this exception caused by an overloaded node?
Hi,

I am running on a cluster where some of the machines are loaded for other
purposes. Occasionally an HBase scan fails with the message below, and I
suspect this is caused by on or more of the region servers being overloaded
with other processes (not hadoop/hbase) and not being able to respond
correctly. Is this possible?

Thanks,

Jon

Exception in thread "main" java.lang.RuntimeException:
org.apache.hadoop.hbase.regionserver.LeaseException:
org.apache.hadoop.hbase.regionserver.LeaseException: lease
'-3573171992963675348' does not exist
    at
org.apache.hadoop.hbase.regionserver.Leases.removeLease(Leases.java:231)
    at
org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:2117)
    at sun.reflect.GeneratedMethodAccessor32.invoke(Unknown Source)
    at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:616)
    at
org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
    at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326)

    at
org.apache.hadoop.hbase.client.HTable$ClientScanner$1.hasNext(HTable.java:1402)
    at prd.Levelize.readPins(Levelize.java:50)
    at prd.Levelize.main(Levelize.java:245)
Caused by: org.apache.hadoop.hbase.regionserver.LeaseException:
org.apache.hadoop.hbase.regionserver.LeaseException: lease
'-3573171992963675348' does not exist
    at
org.apache.hadoop.hbase.regionserver.Leases.removeLease(Leases.java:231)
    at
org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:2117)
    at sun.reflect.GeneratedMethodAccessor32.invoke(Unknown Source)
    at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:616)
    at
org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
    at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326)

    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
    at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:532)
    at
org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:96)
    at
org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:84)
    at
org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:39)
    at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionServerWithRetries(HConnectionManager.java:1325)
    at
org.apache.hadoop.hbase.client.HTable$ClientScanner.next(HTable.java:1293)
    at
org.apache.hadoop.hbase.client.HTable$ClientScanner$1.hasNext(HTable.java:1399)
    ... 2 more
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB