Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase, mail # user - Unbalanced region server


Copy link to this message
-
Re: Unbalanced region server
David Charle 2012-04-24, 04:07
On Mon, Apr 23, 2012 at 8:43 PM, Bryan Beaudreault <[EMAIL PROTECTED]
> wrote:

> Over time the balancer will move around both existing and new regions. A
> few questions: How many tables do you have? Are they all configured
> similarly? What version of hbase are you on?
>
>
70 tables.

hdfs://cluster1.com/hbase 2422000870355

Yes; all configured similarly; and have uniform conf across all regions.

0.20.3 :) don't ask me to upgrade first, and then come back.. :(

balance_switch is not supported in this version.

Is there any other way to split and reclaim the space ?

Thanks for the help
David
> Sent from iPhone.
>
> On Apr 23, 2012, at 11:39 PM, David Charle <[EMAIL PROTECTED]> wrote:
>
> > Does balancer_switch=true; rebalances any skew in the existing keys or it
> > only does for all new regions ? The requests seems to be oK across all
> > servers.
> >
> > On Mon, Apr 23, 2012 at 8:03 PM, David Charle <[EMAIL PROTECTED]>
> wrote:
> >
> >> Thanks Bryan.
> >>
> >> Here is the regions count:
> >>
> >> 1: 2616
> >> 2: 2620
> >> 3: 2623
> >> 4: 2617
> >> 5: 2617
> >>
> >> The skew is in Node 2 where we have the space issue (double the size of
> >> other 4 nodes).
> >>
> >>
> >> On Mon, Apr 23, 2012 at 7:50 PM, Bryan Beaudreault <
> >> [EMAIL PROTECTED]> wrote:
> >>
> >>> Do all nodes have the same number of regions?  If not you may not have
> the
> >>> balancer on.  You can turn it on using balancer_switch true in the
> hbase
> >>> shell.  The balancer also doesn't run if there is a region stuck in
> >>> transition.  In this case if your data is growing rapidly I have seen
> >>> regionservers become lopsided until I could clean up the stuck region
> and
> >>> re-enable the balancer.
> >>>
> >>> Finally, depending on your version of hbase, the balancer only tries to
> >>> keep the number of regions similar across all regionservers.  If you
> have
> >>> tables of different max region sizes I could imagine a case where one
> >>> region server unluckily is hosting more regions from the table with
> larger
> >>> region sizes.  This also might explain the imbalance in requests, if
> one
> >>> of
> >>> your tables gets more traffic than other.  The HMaster UI would be
> helpful
> >>> for determining the spread of regions per table across regionservers.
> >>>
> >>> On Mon, Apr 23, 2012 at 9:41 PM, David Charle <[EMAIL PROTECTED]>
> >>> wrote:
> >>>
> >>>> Hi
> >>>>
> >>>> What can make a region server unbalanced when it comes to space ? (and
> >>>> possibly requests too).
> >>>>
> >>>> For example; I have 5 node cluster (replication factor of 3); and in
> >>> which
> >>>> all 4 has same size; where as one node is almost double in size for
> >>> /hbase.
> >>>>
> >>>> Any help will be appreciated; esp what makes the skew and how to fix
> it
> >>> ?
> >>>>
> >>>> Thanks
> >>>> David
> >>>>
> >>>
> >>
> >>
>