Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
HBase, mail # user - Fixing badly distributed table manually.

David Koch 2012-09-04, 14:56
Pablo Musa 2012-09-04, 15:20
Ted Yu 2012-09-04, 15:12
Guillaume Gardey 2012-09-04, 15:34
David Koch 2012-09-04, 21:42
Copy link to this message
Re: Fixing badly distributed table manually.
Vincent Barat 2012-09-05, 06:37

Balancing regions between RS is correctly handled by HBase : I mean
that your RSs always manage the same number of regions (the balancer
takes care of it).

Unfortunately, balancing all the regions of one particular table
between the RS of your cluster is not always easy, since HBase (as
for 0.90.3) when it comes to splitting a region, create the new one
always on the same RS. This means that if you start with a 1 region
only table, and then you insert lots of data into it, new regions
will always be created to the same RS (if you insert is a M/R job,
you saturate this RS). Eventually, the balancer at a time will
decide to balance one of these regions to other RS, limiting the
issue, but it is not controllable.

Here at Capptain, we solved this problem by developing a special
Python script, based on the HBase shell, allowing to entirely
balance all the regions of all tables to all RS. It ensure that
regions of tables are uniformly deployed on all RS of the cluster,
with a minimum region transitions.

It is fast, and even if it can trigger a lot of region transitions,
there is very few impact at runtime and it can be run safely.

If you are interested, just let me know, I can share it.


Le 04/09/12 23:42, David Koch a �crit :
> Hello,
> Thank you for your replies. We are using CDH4 HBase 0.92. Good call on the
> web interface. The port is blocked so I never really got a chance to test
> it. As far as manual re-balancing is concerned I will check the book.
> /David
> On Tue, Sep 4, 2012 at 5:34 PM, Guillaume Gardey <
>> Hello,
>>> a) What is the easiest way to get an overview of how a table is
>> distributed
>>> across regions of a cluster? I guess I could search .META. but I haven't
>>> figured out how to use filters from shell.
>>> b) What constitutes a "badly distributed" table and how can I re-balance
>>> manually?
>>> c) Is b) needed at all? I know that HBase does its balancing
>> automatically
>>> behind the scenes.
>> I have found that
>> http://bobcopeland.com/blog/2012/04/graphing-hbase-splits/  is a good
>> source of information/tools to look at regions balancing in the cluster and
>> investigate it.
>>> As for a) I tried running this script:
>>> https://github.com/Mendeley/hbase-scripts/blob/master/list_regions.rb
>>> like so:
>>> hbase org.jruby.Main ./list_regions.rb <_my_table>
>>> but I get
>>> ArgumentError: wrong number of arguments (1 for 2)
>>>   (root) at ./list_regions.rb:60
>>> If someone more proficient notices an obvious fix, I'd be glad to hear
>>> about it.
>> Concerning https://github.com/Mendeley/hbase-scripts , I am afraid that
>> this is a repository that is no longer maintained and was written for old
>> releases of hbase (cdh2 I believe). There's no plan to upgrade it to newer
>> releases.
>> Cheers
>> ---
>> Guillaume
Vincent Barat 2012-09-05, 13:19
David Koch 2012-09-05, 08:40
Ivan Balashov 2012-12-24, 16:27