I know there were some changes in .90 in terms of how region balancing
occurs. Is there a resource somewhere that describes the options for the
configuration? Per Jonathan Gray's recommendation we are trying to keep our
region count down to 100 per region server (we are up to 5gb region size).
This has caused us to get a lot of hot spots relative to what we saw with
1000 regions per node. Scanning through #3373 it sounded like there is a
round-robin load balancer we can turn on instead of keeping regions pinned
as I believe they are in .90 by default. We have many tables and certain
region servers seem to get "stuck" with a single table's data in almost
contiguous ranges which really throws a wrench in our load balancing.

Thanks.

In shell is a move command.  You can also force running of balancer
(or even turn it off).

For how the balancer works, in short, it runs every 5 minutes by
default (configurable) and when it runs, using its in-memory notion of
how the cluster is balanced, creates move plans that are immediately
acted upon on balance calculation completion.

Current balancer aims to keep the load of regions even across the cluster.

If you would like to know more beyond whats written above, next up
would be the LoadBalancer classes javadoc:
http://hbase.apache.org/xref/org/apache/hadoop/hbase/master/LoadBalancer.html#42
 Its not bad.  Check it out.

St.Ack

On Wed, Feb 2, 2011 at 7:51 PM, Wayne <[EMAIL PROTECTED]> wrote:
> I know there were some changes in .90 in terms of how region balancing
> occurs. Is there a resource somewhere that describes the options for the
> configuration? Per Jonathan Gray's recommendation we are trying to keep our
> region count down to 100 per region server (we are up to 5gb region size).
> This has caused us to get a lot of hot spots relative to what we saw with
> 1000 regions per node. Scanning through #3373 it sounded like there is a
> round-robin load balancer we can turn on instead of keeping regions pinned
> as I believe they are in .90 by default. We have many tables and certain
> region servers seem to get "stuck" with a single table's data in almost
> contiguous ranges which really throws a wrench in our load balancing.
>
> Thanks.
>

The regions counts are the same per region server which is good. My problem
is that I have 5 tables and several region servers only serve 1 table's
regions. I would like to round robin and scatter all tables across all
region servers. Basically the distribution is not round-robin enough.
Manually moving it is not going to help me. Frankly this goes against the
concept of bigger/less regions. Given what I am seeing without an
alternative I will reduce the max size of the regions and once I get into
the 100s of regions per region server this problem will be resolved. Less
regions is dangerous in terms of avoiding hot spots.

Is there a way to turn off the memory across restarts of where a region
lives? This might help re-balance from scratch.

Thanks.

On Wed, Feb 2, 2011 at 3:17 PM, Stack <[EMAIL PROTECTED]> wrote:

> In shell is a move command.  You can also force running of balancer
> (or even turn it off).
>
> For how the balancer works, in short, it runs every 5 minutes by
> default (configurable) and when it runs, using its in-memory notion of
> how the cluster is balanced, creates move plans that are immediately
> acted upon on balance calculation completion.
>
> Current balancer aims to keep the load of regions even across the cluster.
>
> If you would like to know more beyond whats written above, next up
> would be the LoadBalancer classes javadoc:
>
> http://hbase.apache.org/xref/org/apache/hadoop/hbase/master/LoadBalancer.html#42
>  Its not bad.  Check it out.
>
> St.Ack
>
> On Wed, Feb 2, 2011 at 7:51 PM, Wayne <[EMAIL PROTECTED]> wrote:
> > I know there were some changes in .90 in terms of how region balancing
> > occurs. Is there a resource somewhere that describes the options for the
> > configuration? Per Jonathan Gray's recommendation we are trying to keep
> our
> > region count down to 100 per region server (we are up to 5gb region
> size).
> > This has caused us to get a lot of hot spots relative to what we saw with
> > 1000 regions per node. Scanning through #3373 it sounded like there is a
> > round-robin load balancer we can turn on instead of keeping regions
> pinned
> > as I believe they are in .90 by default. We have many tables and certain
> > region servers seem to get "stuck" with a single table's data in almost
> > contiguous ranges which really throws a wrench in our load balancing.
> >
> > Thanks.
> >
>

On Wed, Feb 2, 2011 at 8:41 PM, Wayne <[EMAIL PROTECTED]> wrote:
> The regions counts are the same per region server which is good. My problem
> is that I have 5 tables and several region servers only serve 1 table's
> regions.

I wonder if this an effect of our deploying splits to same server as
split parent?  Once the phenomeon goes into effect, we'll not break it
that I can see (on restart, we try our best in 0.90.0 to redeploy
regions to where they were running pre-shutdown so we don't lose
locality).

> I would like to round robin and scatter all tables across all
> region servers. Basically the distribution is not round-robin enough.
> Manually moving it is not going to help me. Frankly this goes against the
> concept of bigger/less regions. Given what I am seeing without an
> alternative I will reduce the max size of the regions and once I get into
> the 100s of regions per region server this problem will be resolved. Less
> regions is dangerous in terms of avoiding hot spots.
>
> Is there a way to turn off the memory across restarts of where a region
> lives? This might help re-balance from scratch.
>

>From AssignmentManager:

1226     // Determine what type of assignment to do on startup
1227     boolean retainAssignment = master.getConfiguration().
1228       getBoolean("hbase.master.startup.retainassign", true);

It looks like you could set the above flag in your hbase-site.xml to
false and that should do it (its on by default).

You could knock a few of your regionservers out of the cluster.  Wait
till regions assigned elsewhere, then bring them back up again.  Force
running of assignment.  That might mess stuff up enough?

St.Ack

hbase.master.startup.retainassign=false works like a charm. After a restart
all tables are scattered across all region servers.

Thanks!

On Wed, Feb 2, 2011 at 4:06 PM, Stack <[EMAIL PROTECTED]> wrote:

> On Wed, Feb 2, 2011 at 8:41 PM, Wayne <[EMAIL PROTECTED]> wrote:
> > The regions counts are the same per region server which is good. My
> problem
> > is that I have 5 tables and several region servers only serve 1 table's
> > regions.
>
> I wonder if this an effect of our deploying splits to same server as
> split parent?  Once the phenomeon goes into effect, we'll not break it
> that I can see (on restart, we try our best in 0.90.0 to redeploy
> regions to where they were running pre-shutdown so we don't lose
> locality).
>
> > I would like to round robin and scatter all tables across all
> > region servers. Basically the distribution is not round-robin enough.
> > Manually moving it is not going to help me. Frankly this goes against the
> > concept of bigger/less regions. Given what I am seeing without an
> > alternative I will reduce the max size of the regions and once I get into
> > the 100s of regions per region server this problem will be resolved. Less
> > regions is dangerous in terms of avoiding hot spots.
> >
> > Is there a way to turn off the memory across restarts of where a region
> > lives? This might help re-balance from scratch.
> >
>
> From AssignmentManager:
>
> 1226     // Determine what type of assignment to do on startup
> 1227     boolean retainAssignment = master.getConfiguration().
> 1228       getBoolean("hbase.master.startup.retainassign", true);
>
> It looks like you could set the above flag in your hbase-site.xml to
> false and that should do it (its on by default).
>
> You could knock a few of your regionservers out of the cluster.  Wait
> till regions assigned elsewhere, then bring them back up again.  Force
> running of assignment.  That might mess stuff up enough?
>
> St.Ack
>

I have small question, is there any method to get region
numberOfRequests, master server have
server.getLoad().getNumberOfRequests() but i cannot find any similar to
region?

On 02.02.2011 22:06, Stack wrote:
> On Wed, Feb 2, 2011 at 8:41 PM, Wayne<[EMAIL PROTECTED]>  wrote:
>> The regions counts are the same per region server which is good. My problem
>> is that I have 5 tables and several region servers only serve 1 table's
>> regions.
> I wonder if this an effect of our deploying splits to same server as
> split parent?  Once the phenomeon goes into effect, we'll not break it
> that I can see (on restart, we try our best in 0.90.0 to redeploy
> regions to where they were running pre-shutdown so we don't lose
> locality).
>
>> I would like to round robin and scatter all tables across all
>> region servers. Basically the distribution is not round-robin enough.
>> Manually moving it is not going to help me. Frankly this goes against the
>> concept of bigger/less regions. Given what I am seeing without an
>> alternative I will reduce the max size of the regions and once I get into
>> the 100s of regions per region server this problem will be resolved. Less
>> regions is dangerous in terms of avoiding hot spots.
>>
>> Is there a way to turn off the memory across restarts of where a region
>> lives? This might help re-balance from scratch.
>>
> > From AssignmentManager:
>
> 1226     // Determine what type of assignment to do on startup
> 1227     boolean retainAssignment = master.getConfiguration().
> 1228       getBoolean("hbase.master.startup.retainassign", true);
>
> It looks like you could set the above flag in your hbase-site.xml to
> false and that should do it (its on by default).
>
> You could knock a few of your regionservers out of the cluster.  Wait
> till regions assigned elsewhere, then bring them back up again.  Force
> running of assignment.  That might mess stuff up enough?
>
> St.Ack
>
--

Pozdrawiam
Sebastian Bauer
-----------------------------------------------------
http://tikecik.pl