Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # user >> Region is in transition, in CLOSING state


Copy link to this message
-
Region is in transition, in CLOSING state
Hi,

We are using hbase-0.90.4, and after one of our region servers went down
master reported that one region is in transition, in CLOSING state:

INFO org.apache.hadoop.hbase.master.AssignmentManager: Regions in
transition timed out:  access_tokens,,1312452518317.1878726708
state=CLOSING, ts=1328975812113
INFO org.apache.hadoop.hbase.master.AssignmentManager: Region has been
CLOSING for too long, this should eventually complete or the server will
expire, doing nothing

Region server that went down was hosting 8 regions and all
were successfully closed and  reassigned to  other region servers.
Problem appeared when master tried to rebalance cluster by moving regions
from overloaded servers to less loaded servers:

INFO org.apache.hadoop.hbase.master.LoadBalancer: Calculated a load balance
in 0ms. Moving 3 regions off of 2 overloaded servers onto 2 less loaded
servers

And again 2 regions were successfully moved to less loaded servers
but access_tokens,,1312452518317.1878726708 region stucked in CLOSING
state. We resolved issue by restarting cluster (we had to kill master).

My question is what is best approach in situations like this ? (i have
tried manual unassign from shell did not work)
I have seen few bugs that are addressing similar issue, is this fixed in
hbase-0.92?

Regards
Samir
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB