Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # user >> Unbalanced region server


Copy link to this message
-
Re: Unbalanced region server
Unfortunately I don't have any experience with that version. You might try an hdfs rebalance, but I'm not sure of the implications of that for hbase.  Hopefully someone else can be of more help. Good luck!

Sent from iPhone.

On Apr 24, 2012, at 12:07 AM, David Charle <[EMAIL PROTECTED]> wrote:

> On Mon, Apr 23, 2012 at 8:43 PM, Bryan Beaudreault <[EMAIL PROTECTED]
>> wrote:
>
>> Over time the balancer will move around both existing and new regions. A
>> few questions: How many tables do you have? Are they all configured
>> similarly? What version of hbase are you on?
>>
>>
> 70 tables.
>
> hdfs://cluster1.com/hbase 2422000870355
>
> Yes; all configured similarly; and have uniform conf across all regions.
>
> 0.20.3 :) don't ask me to upgrade first, and then come back.. :(
>
> balance_switch is not supported in this version.
>
> Is there any other way to split and reclaim the space ?
>
> Thanks for the help
> David
>
>
>> Sent from iPhone.
>>
>> On Apr 23, 2012, at 11:39 PM, David Charle <[EMAIL PROTECTED]> wrote:
>>
>>> Does balancer_switch=true; rebalances any skew in the existing keys or it
>>> only does for all new regions ? The requests seems to be oK across all
>>> servers.
>>>
>>> On Mon, Apr 23, 2012 at 8:03 PM, David Charle <[EMAIL PROTECTED]>
>> wrote:
>>>
>>>> Thanks Bryan.
>>>>
>>>> Here is the regions count:
>>>>
>>>> 1: 2616
>>>> 2: 2620
>>>> 3: 2623
>>>> 4: 2617
>>>> 5: 2617
>>>>
>>>> The skew is in Node 2 where we have the space issue (double the size of
>>>> other 4 nodes).
>>>>
>>>>
>>>> On Mon, Apr 23, 2012 at 7:50 PM, Bryan Beaudreault <
>>>> [EMAIL PROTECTED]> wrote:
>>>>
>>>>> Do all nodes have the same number of regions?  If not you may not have
>> the
>>>>> balancer on.  You can turn it on using balancer_switch true in the
>> hbase
>>>>> shell.  The balancer also doesn't run if there is a region stuck in
>>>>> transition.  In this case if your data is growing rapidly I have seen
>>>>> regionservers become lopsided until I could clean up the stuck region
>> and
>>>>> re-enable the balancer.
>>>>>
>>>>> Finally, depending on your version of hbase, the balancer only tries to
>>>>> keep the number of regions similar across all regionservers.  If you
>> have
>>>>> tables of different max region sizes I could imagine a case where one
>>>>> region server unluckily is hosting more regions from the table with
>> larger
>>>>> region sizes.  This also might explain the imbalance in requests, if
>> one
>>>>> of
>>>>> your tables gets more traffic than other.  The HMaster UI would be
>> helpful
>>>>> for determining the spread of regions per table across regionservers.
>>>>>
>>>>> On Mon, Apr 23, 2012 at 9:41 PM, David Charle <[EMAIL PROTECTED]>
>>>>> wrote:
>>>>>
>>>>>> Hi
>>>>>>
>>>>>> What can make a region server unbalanced when it comes to space ? (and
>>>>>> possibly requests too).
>>>>>>
>>>>>> For example; I have 5 node cluster (replication factor of 3); and in
>>>>> which
>>>>>> all 4 has same size; where as one node is almost double in size for
>>>>> /hbase.
>>>>>>
>>>>>> Any help will be appreciated; esp what makes the skew and how to fix
>> it
>>>>> ?
>>>>>>
>>>>>> Thanks
>>>>>> David
>>>>>>
>>>>>
>>>>
>>>>
>>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB