Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
HBase >> mail # user >> Re: Region has been CLOSING for too long, this should eventually complete or the server will expire, send RPC again


+
Ted Yu 2013-04-16, 01:54
Copy link to this message
-
Re: Region has been CLOSING for too long, this should eventually complete or the server will expire, send RPC again
Hi Dylan,

$HBASE_HOME/bin/hbase hbck -fix

the above command can bring back the cluster to normal state.

Just if master restarted while ROOT/META region in closing(during move or
balance) then the problem you have reported will come easily. Thanks for
detailed logs.

Raised an issue for this. You can follow HBASE-8353.
<https://issues.apache.org/jira/browse/HBASE-8353>
On Tue, Apr 16, 2013 at 7:24 AM, Ted Yu <[EMAIL PROTECTED]> wrote:

> I think this question would be more appropriate for HBase user mailing
> list.
>
> Moving hadoop user to bcc.
>
> Please tell us the HBase version you are using.
>
> Thanks
>
> On Mon, Apr 15, 2013 at 6:51 PM, dylan <[EMAIL PROTECTED]> wrote:
>
> > Hi****
> >
> > ** **
> >
> > I am a newer for hadoop, and set up hadoop with tarball . I have 5 nodes
> > for cluster, 2 NN nodes with QJM (3 Journal Nodes, one of them on DN
> node.
> >  ), 3 DN nodes with zookeepers,  It works fine.  When I reboot one data
> > node machine which includes zookeeper, after that , restart all
> processes.
> > The hadoop works fine, but hbase not. I cannot disable tables and drop
> > tables.****
> >
> > ** **
> >
> > The logs an follows:****
> >
> > The Hbase HMaster log:****
> >
> > DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Attempted to
> > unassign region -ROOT-,,0.70236052 but it is not currently assigned
> anywhere
> > ****
> >
> > ,683 INFO org.apache.hadoop.hbase.master.AssignmentManager: Regions in
> > transition timed out:  -ROOT-,,0.70236052 state=CLOSING,
> ts=1366001558865,
> > server=Master,60000,1366001238313****
> >
> > ,683 INFO org.apache.hadoop.hbase.master.AssignmentManager: Region has
> > been CLOSING for too long, this should eventually complete or the server
> > will expire, send RPC again****
> >
> > 10,684 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Starting
> > unassignment of region -ROOT-,,0.70236052 (offlining)****
> >
> > ** **
> >
> > The Hbase HRegionServer log:****
> >
> > ** **
> >
> > DEBUG org.apache.hadoop.hbase.io.hfile.LruBlockCache: Stats: total=7.44
> > MB, free=898.81 MB, max=906.24 MB, blocks=0, accesses=0, hits=0,
> > hitRatio=0, cachingAccesses=0, cachingHits=0, cachingHitsRatio=0,
> > evictions=0, evicted=0, evictedPerRun=NaN****
> >
> > ** **
> >
> > The Hbase Web show:****
> >
> > Region                                              State****
> >
> > 70236052    -ROOT-,,0.70236052 state=CLOSING, ts=Mon Apr 15 12:52:38 CST
> > 2013 (75440s ago), server=Master,60000,1366001238313****
> >
> > ** **
> >
> > How fix it?****
> >
> > ** **
> >
> > Thanks.****
> >
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB