Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
HBase >> mail # user >> Unbalanced region server


+
David Charle 2012-04-24, 01:41
+
Bryan Beaudreault 2012-04-24, 02:50
+
David Charle 2012-04-24, 03:03
+
David Charle 2012-04-24, 03:39
+
Bryan Beaudreault 2012-04-24, 03:43
Copy link to this message
-
Re: Unbalanced region server
On Mon, Apr 23, 2012 at 8:43 PM, Bryan Beaudreault <[EMAIL PROTECTED]
> wrote:

> Over time the balancer will move around both existing and new regions. A
> few questions: How many tables do you have? Are they all configured
> similarly? What version of hbase are you on?
>
>
70 tables.

hdfs://cluster1.com/hbase 2422000870355

Yes; all configured similarly; and have uniform conf across all regions.

0.20.3 :) don't ask me to upgrade first, and then come back.. :(

balance_switch is not supported in this version.

Is there any other way to split and reclaim the space ?

Thanks for the help
David
> Sent from iPhone.
>
> On Apr 23, 2012, at 11:39 PM, David Charle <[EMAIL PROTECTED]> wrote:
>
> > Does balancer_switch=true; rebalances any skew in the existing keys or it
> > only does for all new regions ? The requests seems to be oK across all
> > servers.
> >
> > On Mon, Apr 23, 2012 at 8:03 PM, David Charle <[EMAIL PROTECTED]>
> wrote:
> >
> >> Thanks Bryan.
> >>
> >> Here is the regions count:
> >>
> >> 1: 2616
> >> 2: 2620
> >> 3: 2623
> >> 4: 2617
> >> 5: 2617
> >>
> >> The skew is in Node 2 where we have the space issue (double the size of
> >> other 4 nodes).
> >>
> >>
> >> On Mon, Apr 23, 2012 at 7:50 PM, Bryan Beaudreault <
> >> [EMAIL PROTECTED]> wrote:
> >>
> >>> Do all nodes have the same number of regions?  If not you may not have
> the
> >>> balancer on.  You can turn it on using balancer_switch true in the
> hbase
> >>> shell.  The balancer also doesn't run if there is a region stuck in
> >>> transition.  In this case if your data is growing rapidly I have seen
> >>> regionservers become lopsided until I could clean up the stuck region
> and
> >>> re-enable the balancer.
> >>>
> >>> Finally, depending on your version of hbase, the balancer only tries to
> >>> keep the number of regions similar across all regionservers.  If you
> have
> >>> tables of different max region sizes I could imagine a case where one
> >>> region server unluckily is hosting more regions from the table with
> larger
> >>> region sizes.  This also might explain the imbalance in requests, if
> one
> >>> of
> >>> your tables gets more traffic than other.  The HMaster UI would be
> helpful
> >>> for determining the spread of regions per table across regionservers.
> >>>
> >>> On Mon, Apr 23, 2012 at 9:41 PM, David Charle <[EMAIL PROTECTED]>
> >>> wrote:
> >>>
> >>>> Hi
> >>>>
> >>>> What can make a region server unbalanced when it comes to space ? (and
> >>>> possibly requests too).
> >>>>
> >>>> For example; I have 5 node cluster (replication factor of 3); and in
> >>> which
> >>>> all 4 has same size; where as one node is almost double in size for
> >>> /hbase.
> >>>>
> >>>> Any help will be appreciated; esp what makes the skew and how to fix
> it
> >>> ?
> >>>>
> >>>> Thanks
> >>>> David
> >>>>
> >>>
> >>
> >>
>
+
Bryan Beaudreault 2012-04-24, 04:14
+
Stack 2012-04-24, 04:13
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB