Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
HBase >> mail # user >> Re: Region has been CLOSING for too long, this should eventually complete or the server will expire, send RPC again


+
Ted Yu 2013-04-16, 01:54
Copy link to this message
-
Re: Region has been CLOSING for too long, this should eventually complete or the server will expire, send RPC again
Hi Dylan,

$HBASE_HOME/bin/hbase hbck -fix

the above command can bring back the cluster to normal state.

Just if master restarted while ROOT/META region in closing(during move or
balance) then the problem you have reported will come easily. Thanks for
detailed logs.

Raised an issue for this. You can follow HBASE-8353.
<https://issues.apache.org/jira/browse/HBASE-8353>
On Tue, Apr 16, 2013 at 7:24 AM, Ted Yu <[EMAIL PROTECTED]> wrote:

> I think this question would be more appropriate for HBase user mailing
> list.
>
> Moving hadoop user to bcc.
>
> Please tell us the HBase version you are using.
>
> Thanks
>
> On Mon, Apr 15, 2013 at 6:51 PM, dylan <[EMAIL PROTECTED]> wrote:
>
> > Hi****
> >
> > ** **
> >
> > I am a newer for hadoop, and set up hadoop with tarball . I have 5 nodes
> > for cluster, 2 NN nodes with QJM (3 Journal Nodes, one of them on DN
> node.
> >  ), 3 DN nodes with zookeepers,  It works fine.  When I reboot one data
> > node machine which includes zookeeper, after that , restart all
> processes.
> > The hadoop works fine, but hbase not. I cannot disable tables and drop
> > tables.****
> >
> > ** **
> >
> > The logs an follows:****
> >
> > The Hbase HMaster log:****
> >
> > DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Attempted to
> > unassign region -ROOT-,,0.70236052 but it is not currently assigned
> anywhere
> > ****
> >
> > ,683 INFO org.apache.hadoop.hbase.master.AssignmentManager: Regions in
> > transition timed out:  -ROOT-,,0.70236052 state=CLOSING,
> ts=1366001558865,
> > server=Master,60000,1366001238313****
> >
> > ,683 INFO org.apache.hadoop.hbase.master.AssignmentManager: Region has
> > been CLOSING for too long, this should eventually complete or the server
> > will expire, send RPC again****
> >
> > 10,684 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Starting
> > unassignment of region -ROOT-,,0.70236052 (offlining)****
> >
> > ** **
> >
> > The Hbase HRegionServer log:****
> >
> > ** **
> >
> > DEBUG org.apache.hadoop.hbase.io.hfile.LruBlockCache: Stats: total=7.44
> > MB, free=898.81 MB, max=906.24 MB, blocks=0, accesses=0, hits=0,
> > hitRatio=0, cachingAccesses=0, cachingHits=0, cachingHitsRatio=0,
> > evictions=0, evicted=0, evictedPerRun=NaN****
> >
> > ** **
> >
> > The Hbase Web show:****
> >
> > Region                                              State****
> >
> > 70236052    -ROOT-,,0.70236052 state=CLOSING, ts=Mon Apr 15 12:52:38 CST
> > 2013 (75440s ago), server=Master,60000,1366001238313****
> >
> > ** **
> >
> > How fix it?****
> >
> > ** **
> >
> > Thanks.****
> >
>