Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
HBase, mail # user - Invalid HTable(Pool) instance after Master Failover


Copy link to this message
-
Invalid HTable(Pool) instance after Master Failover
Micah Whitacre 2012-06-13, 21:08
We are currently running with 0.92.0 (cdh4b1)  and have configured HA
so that the master will fail over automatically (we will be upgrading
to 0.92.1 cdh4u0 soon).  In testing failover we've noticed that in our
long running processes which hold onto an instance of HTable or
HTablePool when the master fails over our long running processes fail
when interacting with HBase because of exceptions like the following.
Should those instances be resilient to failover?  I would at least
expect for HTablePool to invalidate bad pooled tables.  Is there a way
to do this or a better way than catching RetriesExhaustedException and
invalidating the pools ourself?  Or is holding the HTablePool instance
for a long time an anti-pattern?

Caused by: org.apache.hadoop.hbase.client.RetriesExhaustedException:
Failed after attempts=10, exceptions:
Mon Jun 11 14:04:26 CDT 2012, org.apache.hadoop.hbase.client.HTable
$5@72be6280, java.io.IOException:
org.apache.hadoop.hbase.client.HConnectionManager
$HConnectionImplementation@430a14ad closed
Mon Jun 11 14:04:27 CDT 2012, org.apache.hadoop.hbase.client.HTable
$5@72be6280, java.io.IOException:
org.apache.hadoop.hbase.client.HConnectionManager
$HConnectionImplementation@430a14ad closed
Mon Jun 11 14:04:28 CDT 2012, org.apache.hadoop.hbase.client.HTable
$5@72be6280, java.io.IOException:
org.apache.hadoop.hbase.client.HConnectionManager
$HConnectionImplementation@430a14ad closed
Mon Jun 11 14:04:29 CDT 2012, org.apache.hadoop.hbase.client.HTable
$5@72be6280, java.io.IOException:
org.apache.hadoop.hbase.client.HConnectionManager
$HConnectionImplementation@430a14ad closed
Mon Jun 11 14:04:31 CDT 2012, org.apache.hadoop.hbase.client.HTable
$5@72be6280, java.io.IOException:
org.apache.hadoop.hbase.client.HConnectionManager
$HConnectionImplementation@430a14ad closed
Mon Jun 11 14:04:33 CDT 2012, org.apache.hadoop.hbase.client.HTable
$5@72be6280, java.io.IOException:
org.apache.hadoop.hbase.client.HConnectionManager
$HConnectionImplementation@430a14ad closed
Mon Jun 11 14:04:37 CDT 2012, org.apache.hadoop.hbase.client.HTable
$5@72be6280, java.io.IOException:
org.apache.hadoop.hbase.client.HConnectionManager
$HConnectionImplementation@430a14ad closed
Mon Jun 11 14:04:41 CDT 2012, org.apache.hadoop.hbase.client.HTable
$5@72be6280, java.io.IOException:
org.apache.hadoop.hbase.client.HConnectionManager
$HConnectionImplementation@430a14ad closed
Mon Jun 11 14:04:49 CDT 2012, org.apache.hadoop.hbase.client.HTable
$5@72be6280, java.io.IOException:
org.apache.hadoop.hbase.client.HConnectionManager
$HConnectionImplementation@430a14ad closed
Mon Jun 11 14:05:05 CDT 2012, org.apache.hadoop.hbase.client.HTable
$5@72be6280, java.io.IOException:
org.apache.hadoop.hbase.client.HConnectionManager
$HConnectionImplementation@430a14ad closed

at org.apache.hadoop.hbase.client.HConnectionManager
$HConnectionImplementation.getRegionServerWithRetries(HConnectionManager.java:
1345)
at org.apache.hadoop.hbase.client.HTable.get(HTable.java:657)

Thanks,
Micah