We upgraded to CDH4.1.2 (contains HBASE-3373) in one of our
environments.  After filling that environment with data, I was
surprised to see that regions were not balanced across regionservers
at the table level.  We have restarted all regionservers at least once
here.

In [1], I see Stack's reference to temporarily adding
"hbase.master.startup.retainassign=false" and restarting nodes.  Is
this a necessary step on the path to region balancing nirvana?

Norbert

[1] http://search-hadoop.com/m/MQSPEyUQIv1

Can you give us a little more detail on how much deviation the region
counts on region servers have ?

There is a parameter, hbase.regions.slop, with default value of 0.2
This parameter allows region count to deviate by certain percentage from
average region count.

You can tighten the value for this parameter and see if you get better
results.

I will also put the above summary on HBASE-3373.

Thanks

On Tue, Dec 4, 2012 at 8:42 AM, Norbert Burger <[EMAIL PROTECTED]>wrote:

> We upgraded to CDH4.1.2 (contains HBASE-3373) in one of our
> environments.  After filling that environment with data, I was
> surprised to see that regions were not balanced across regionservers
> at the table level.  We have restarted all regionservers at least once
> here.
>
> In [1], I see Stack's reference to temporarily adding
> "hbase.master.startup.retainassign=false" and restarting nodes.  Is
> this a necessary step on the path to region balancing nirvana?
>
> Norbert
>
> [1] http://search-hadoop.com/m/MQSPEyUQIv1
>

In CDH4.1.2, per table region balancing is turned off by default.

You can change the configuration to turn it on.

Thanks,
Jimmy

On Tue, Dec 4, 2012 at 11:10 AM, Ted Yu <[EMAIL PROTECTED]> wrote:
> Can you give us a little more detail on how much deviation the region
> counts on region servers have ?
>
> There is a parameter, hbase.regions.slop, with default value of 0.2
> This parameter allows region count to deviate by certain percentage from
> average region count.
>
> You can tighten the value for this parameter and see if you get better
> results.
>
> I will also put the above summary on HBASE-3373.
>
> Thanks
>
> On Tue, Dec 4, 2012 at 8:42 AM, Norbert Burger <[EMAIL PROTECTED]>wrote:
>
>> We upgraded to CDH4.1.2 (contains HBASE-3373) in one of our
>> environments.  After filling that environment with data, I was
>> surprised to see that regions were not balanced across regionservers
>> at the table level.  We have restarted all regionservers at least once
>> here.
>>
>> In [1], I see Stack's reference to temporarily adding
>> "hbase.master.startup.retainassign=false" and restarting nodes.  Is
>> this a necessary step on the path to region balancing nirvana?
>>
>> Norbert
>>
>> [1] http://search-hadoop.com/m/MQSPEyUQIv1
>>

Thanks, Jimmy.  Do you mean the config
"hbase.master.loadbalance.bytable"?  According to [1] and [2], it is
"true" by default.

[1] https://issues.apache.org/jira/secure/attachment/12509174/3373.txt
[2] http://search-hadoop.com/m/M6z7G1PKejw

Norbert

On Tue, Dec 4, 2012 at 2:23 PM, Jimmy Xiang <[EMAIL PROTECTED]> wrote:
> In CDH4.1.2, per table region balancing is turned off by default.
>
> You can change the configuration to turn it on.
>
> Thanks,
> Jimmy
>
> On Tue, Dec 4, 2012 at 11:10 AM, Ted Yu <[EMAIL PROTECTED]> wrote:
>> Can you give us a little more detail on how much deviation the region
>> counts on region servers have ?
>>
>> There is a parameter, hbase.regions.slop, with default value of 0.2
>> This parameter allows region count to deviate by certain percentage from
>> average region count.
>>
>> You can tighten the value for this parameter and see if you get better
>> results.
>>
>> I will also put the above summary on HBASE-3373.
>>
>> Thanks
>>
>> On Tue, Dec 4, 2012 at 8:42 AM, Norbert Burger <[EMAIL PROTECTED]>wrote:
>>
>>> We upgraded to CDH4.1.2 (contains HBASE-3373) in one of our
>>> environments.  After filling that environment with data, I was
>>> surprised to see that regions were not balanced across regionservers
>>> at the table level.  We have restarted all regionservers at least once
>>> here.
>>>
>>> In [1], I see Stack's reference to temporarily adding
>>> "hbase.master.startup.retainassign=false" and restarting nodes.  Is
>>> this a necessary step on the path to region balancing nirvana?
>>>
>>> Norbert
>>>
>>> [1] http://search-hadoop.com/m/MQSPEyUQIv1
>>>

On Tue, Dec 4, 2012 at 2:10 PM, Ted Yu <[EMAIL PROTECTED]> wrote:
> Can you give us a little more detail on how much deviation the region
> counts on region servers have ?
>
> There is a parameter, hbase.regions.slop, with default value of 0.2
> This parameter allows region count to deviate by certain percentage from
> average region count.
>
> You can tighten the value for this parameter and see if you get better
> results.

Thanks for the response.  I think this isn't an issue with region slop
-- I had a table with only ~15 regions, and half of them ended up on
one region server.  I've since manually rebalanced that table, but
here's an example from different table [1].

Norbert

[1]
Regions by Region Server
Region Server Region Count
http://aspen8hdp9.turner.com:60030/ 25
http://aspen8hdp10.turner.com:60030/ 25
http://aspen8hdp16.turner.com:60030/ 38
http://aspen8hdp13.turner.com:60030/ 26
http://aspen8hdp15.turner.com:60030/ 26
http://aspen8hdp12.turner.com:60030/ 43
http://aspen8hdp14.turner.com:60030/ 31
http://aspen8hdp11.turner.com:60030/ 42

Thanks for the information.

Please turn on per-table load balancing and tell us what it looks like.

On Tue, Dec 4, 2012 at 11:32 AM, Norbert Burger <[EMAIL PROTECTED]>wrote:

> On Tue, Dec 4, 2012 at 2:10 PM, Ted Yu <[EMAIL PROTECTED]> wrote:
> > Can you give us a little more detail on how much deviation the region
> > counts on region servers have ?
> >
> > There is a parameter, hbase.regions.slop, with default value of 0.2
> > This parameter allows region count to deviate by certain percentage from
> > average region count.
> >
> > You can tighten the value for this parameter and see if you get better
> > results.
>
> Thanks for the response.  I think this isn't an issue with region slop
> -- I had a table with only ~15 regions, and half of them ended up on
> one region server.  I've since manually rebalanced that table, but
> here's an example from different table [1].
>
> Norbert
>
> [1]
> Regions by Region Server
> Region Server   Region Count
> http://aspen8hdp9.turner.com:60030/     25
> http://aspen8hdp10.turner.com:60030/    25
> http://aspen8hdp16.turner.com:60030/    38
> http://aspen8hdp13.turner.com:60030/    26
> http://aspen8hdp15.turner.com:60030/    26
> http://aspen8hdp12.turner.com:60030/    43
> http://aspen8hdp14.turner.com:60030/    31
> http://aspen8hdp11.turner.com:60030/    42
>

Right, that's config.  You can turn it on and restart the cluster.

Upstream it is on by default.  However, it is turned off by default in
CDH4.1.2 to be backward compatible.

Thanks,
Jimmy

On Tue, Dec 4, 2012 at 11:28 AM, Norbert Burger
<[EMAIL PROTECTED]> wrote:
> Thanks, Jimmy.  Do you mean the config
> "hbase.master.loadbalance.bytable"?  According to [1] and [2], it is
> "true" by default.
>
> [1] https://issues.apache.org/jira/secure/attachment/12509174/3373.txt
> [2] http://search-hadoop.com/m/M6z7G1PKejw
>
> Norbert
>
> On Tue, Dec 4, 2012 at 2:23 PM, Jimmy Xiang <[EMAIL PROTECTED]> wrote:
>> In CDH4.1.2, per table region balancing is turned off by default.
>>
>> You can change the configuration to turn it on.
>>
>> Thanks,
>> Jimmy
>>
>> On Tue, Dec 4, 2012 at 11:10 AM, Ted Yu <[EMAIL PROTECTED]> wrote:
>>> Can you give us a little more detail on how much deviation the region
>>> counts on region servers have ?
>>>
>>> There is a parameter, hbase.regions.slop, with default value of 0.2
>>> This parameter allows region count to deviate by certain percentage from
>>> average region count.
>>>
>>> You can tighten the value for this parameter and see if you get better
>>> results.
>>>
>>> I will also put the above summary on HBASE-3373.
>>>
>>> Thanks
>>>
>>> On Tue, Dec 4, 2012 at 8:42 AM, Norbert Burger <[EMAIL PROTECTED]>wrote:
>>>
>>>> We upgraded to CDH4.1.2 (contains HBASE-3373) in one of our
>>>> environments.  After filling that environment with data, I was
>>>> surprised to see that regions were not balanced across regionservers
>>>> at the table level.  We have restarted all regionservers at least once
>>>> here.
>>>>
>>>> In [1], I see Stack's reference to temporarily adding
>>>> "hbase.master.startup.retainassign=false" and restarting nodes.  Is
>>>> this a necessary step on the path to region balancing nirvana?
>>>>
>>>> Norbert
>>>>
>>>> [1] http://search-hadoop.com/m/MQSPEyUQIv1
>>>>

Forcing hbase.master.loadbalance.bytable=true seems to have resolved
my balance issue.  Thanks again.

Norbert

On Tue, Dec 4, 2012 at 2:56 PM, Jimmy Xiang <[EMAIL PROTECTED]> wrote:
> Right, that's config.  You can turn it on and restart the cluster.
>
> Upstream it is on by default.  However, it is turned off by default in
> CDH4.1.2 to be backward compatible.
>
> Thanks,
> Jimmy
>
> On Tue, Dec 4, 2012 at 11:28 AM, Norbert Burger
> <[EMAIL PROTECTED]> wrote:
>> Thanks, Jimmy.  Do you mean the config
>> "hbase.master.loadbalance.bytable"?  According to [1] and [2], it is
>> "true" by default.
>>
>> [1] https://issues.apache.org/jira/secure/attachment/12509174/3373.txt
>> [2] http://search-hadoop.com/m/M6z7G1PKejw
>>
>> Norbert
>>
>> On Tue, Dec 4, 2012 at 2:23 PM, Jimmy Xiang <[EMAIL PROTECTED]> wrote:
>>> In CDH4.1.2, per table region balancing is turned off by default.
>>>
>>> You can change the configuration to turn it on.
>>>
>>> Thanks,
>>> Jimmy
>>>
>>> On Tue, Dec 4, 2012 at 11:10 AM, Ted Yu <[EMAIL PROTECTED]> wrote:
>>>> Can you give us a little more detail on how much deviation the region
>>>> counts on region servers have ?
>>>>
>>>> There is a parameter, hbase.regions.slop, with default value of 0.2
>>>> This parameter allows region count to deviate by certain percentage from
>>>> average region count.
>>>>
>>>> You can tighten the value for this parameter and see if you get better
>>>> results.
>>>>
>>>> I will also put the above summary on HBASE-3373.
>>>>
>>>> Thanks
>>>>
>>>> On Tue, Dec 4, 2012 at 8:42 AM, Norbert Burger <[EMAIL PROTECTED]>wrote:
>>>>
>>>>> We upgraded to CDH4.1.2 (contains HBASE-3373) in one of our
>>>>> environments.  After filling that environment with data, I was
>>>>> surprised to see that regions were not balanced across regionservers
>>>>> at the table level.  We have restarted all regionservers at least once
>>>>> here.
>>>>>
>>>>> In [1], I see Stack's reference to temporarily adding
>>>>> "hbase.master.startup.retainassign=false" and restarting nodes.  Is
>>>>> this a necessary step on the path to region balancing nirvana?
>>>>>
>>>>> Norbert
>>>>>
>>>>> [1] http://search-hadoop.com/m/MQSPEyUQIv1
>>>>>