Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
HBase, mail # user - [Stand alone - distributed mode] HBase master isn't initializing completely


Copy link to this message
-
[Stand alone - distributed mode] HBase master isn't initializing completely
techbuddy 2012-10-11, 21:11
Hi,

I've a standalone Hbase cluster configured in a distributed mode (i.e. the
ZK, Master and the RegServer all running in separate JVMs on the same host).

The HBase master doesn't seem to be intializing successfully. This has
started happening since I encountered the stop-hbase script going into an
indefinite loop followed by a forced kill of the three Hbase processes
(Zk,Master and Region server). I did manually kill these processes couple of
times earlier too, but the Hbase master never stalled partially initialized.
I have also removed the zookeeper and the hbase/test/hbase folders in an
effort to clean things up before triggering the start-hbase script.

Symptoms

1. The create table command on the Hbase shell errors out with a
PleaseHoldException (Master in initializing)

2. On the Admin UI, the regon server lists only the ROOT and the single META
regions. It doesn't list one single region of the user tables. However ,a
list command on the shell indeed lists all the user tables (that were
created in the past)

3. On the Master logs, I see the following locate region failures and
retries on the META table

2012-10-11 14:09:03,970 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
locateRegionInMeta parentTable=-ROOT-,
metaLocation={region=-ROOT-,,0.70236052,
hostname=materialstranger.corp.XXX.com, port=60020}, attempt=77 of 100
failed; retrying after sleep of 32172 because: Connection refused
2012-10-11 14:09:03,971 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
Looked up root region location,
connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@1bbbafc;
serverName=materialstranger.corp.XXX.com,60020,1349987514876
2012-10-11 14:09:36,197 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
Looked up root region location,
connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@1bbbafc;
serverName=materialstranger.corp.XXX.com,60020,1349987514876
2012-10-11 14:09:36,197 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
locateRegionInMeta parentTable=-ROOT-,
metaLocation={region=-ROOT-,,0.70236052,
hostname=materialstranger.corp.XXX.com, port=60020}, attempt=78 of 100
failed; retrying after sleep of 32275 because: Connection refused
2012-10-11 14:09:36,198 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
Looked up root region location,
connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@1bbbafc;
serverName=materialstranger.corp.XXX.com,60020,1349987514876
Any pointers on how to restore the cluster to a clean ,working state would
be highly appreciated.

-SB
--
View this message in context: http://apache-hbase.679495.n3.nabble.com/Stand-alone-distributed-mode-HBase-master-isn-t-initializing-completely-tp4032869.html
Sent from the HBase User mailing list archive at Nabble.com.
+
谢良 2012-10-12, 04:00
+
techbuddy 2012-10-12, 07:27
+
Suraj Varma 2012-10-12, 07:47
+
abhishek1015 2013-06-30, 04:59
+
techbuddy 2013-06-30, 06:09