Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
HBase, mail # user - Unable to fully decommission a RS


+
Yves Langisch 2013-03-06, 08:17
Copy link to this message
-
Re: Unable to fully decommission a RS
Nicolas Liochon 2013-03-06, 08:50
Yes, decommissioning the regionserver does not mean decommissioning the
datanode.
Here, if I understand well your first step, you migrated the regions to
other regions servers. Physically, the data was still on the previous
machine, with the hdfs datanode. It's not used anymore for writes if all
other RS have a local DN, but it's still used for reads.  A replication
factor on 1 makes things tricky. Running a major compaction in HBase should
be enough to make files local. After this, you then want to decommission
the datanode in hdfs. Wait for the replication to be finished (if there is
anything to replicate). Then you're done.

So it would be:
1) decommission RS
2) stop RS
3) major compaction in HBase
4) decommission DN
5) stop DN
6) Shutdown box

Nicolas
On Wed, Mar 6, 2013 at 9:17 AM, Yves Langisch <[EMAIL PROTECTED]> wrote:

> Hi,
>
> I'm trying to decommission a RS, i.e. migrate all regions to different
> nodes in order to shutdown that specific node. First I've moved all regions
> to different nodes and made sure that there are no regions anymore on this
> node:
>
> ...
>     ip-11-11-111-111.ec2.internal:**60020 1362060123906
>         requestsPerSecond=0, numberOfOnlineRegions=0, usedHeapMB=118,
> maxHeapMB=9974
> ...
>
> After moving the regions I've stopped the HBase and Hadoop services on the
> decommissioned. I though that's it but now when accessing the moved regions
> on the new nodes I get error messages in the logs that blocks are not being
> found...On the decommissioned node I can still see many blk_* files in the
> HDFS. Is there anything else that needs to be done to fully transfer all
> region data to the new nodes? It looks like the regions are transferred but
> the related data is still on the old node...Maybe the replication factor of
> only 1 (I know....) is the problem?!?
>
> I'm using HBase 0.92.0 btw.
>
> Thanks
> Yves
>