Hi,

Lars blog (1) mentions that data locality for the region servers is lost
when HBase cluster is restarted. It's also mentioned at the end that work
is going in HBase to assign regions to RS taking data locality into
consideration. The blog entry is 18 months old and so I would like to know
if this has been incorporated into the latest HBase release or data
locality is lost till a compaction is complete.

Regards,
Praveen

(1) - http://www.larsgeorge.com/2010/05/hbase-file-locality-in-hdfs.html

Hi,

On Tue, Feb 14, 2012 at 7:13 AM, Praveen Sripati
<[EMAIL PROTECTED]> wrote:
> Lars blog (1) mentions that data locality for the region servers is lost
> when HBase cluster is restarted. It's also mentioned at the end that work
> is going in HBase to assign regions to RS taking data locality into
> consideration. The blog entry is 18 months old and so I would like to know
> if this has been incorporated into the latest HBase release or data
> locality is lost till a compaction is complete.

JIRA is down for me, but here is the JIRA:

https://issues.apache.org/jira/browse/HBASE-2896

I am pretty sure it's been included in the latest HBase release as it's in CDH3.

Brock

--
Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/

Region allocation is kept in the next restart (
https://issues.apache.org/jira/browse/HBASE-2896 ). This is also present in
the CDH3 code.
Nevertheless if you have a server that did not start correctly you will
have region that will move from it and locality will not remain (even after
you start the problematic node, since he will get random regions)
The best solution would be effectivly
https://issues.apache.org/jira/browse/HBASE-57
Mikael.S

On Tue, Feb 14, 2012 at 3:19 PM, Brock Noland <[EMAIL PROTECTED]> wrote:

> Hi,
>
> On Tue, Feb 14, 2012 at 7:13 AM, Praveen Sripati
> <[EMAIL PROTECTED]> wrote:
> > Lars blog (1) mentions that data locality for the region servers is lost
> > when HBase cluster is restarted. It's also mentioned at the end that work
> > is going in HBase to assign regions to RS taking data locality into
> > consideration. The blog entry is 18 months old and so I would like to
> know
> > if this has been incorporated into the latest HBase release or data
> > locality is lost till a compaction is complete.
>
> JIRA is down for me, but here is the JIRA:
>
> https://issues.apache.org/jira/browse/HBASE-2896
>
> I am pretty sure it's been included in the latest HBase release as it's in
> CDH3.
>
> Brock
>
> --
> Apache MRUnit - Unit testing MapReduce -
> http://incubator.apache.org/mrunit/
>

--
Mikael.S

On Mon, Feb 20, 2012 at 5:03 AM, Praveen Sripati
<[EMAIL PROTECTED]> wrote:
> It would be nice to consider
> both the resource usage of the region and the data locality into
> consideration, not just purely based on the number of regions in the region
> server as implemented currently.
>

Yes.

> The file to block mapping can be found from the HDFS NameNode, but how to
> find out which regions are loaded (# of requests, cpu and memory
> perspective) and which are not? I could not see any resource utilization in
> the region server pages.
>

In 0.92 there is hits per region and this gets reported to the master
as part of ClusterStatus as does memory usage.  This could be factored
into a new balance algorithm.  Could also send over cpu and hardware
profile for factoring (though much of this is available via JMX --
either we get these into clusterstatus or master does poll on jmx
after it sees new server to get server profile)

> Also, curious if HBASE-57 makes sense, since the major compaction runs
> every 24 hrs

Its recommended that you run major compactions yourself at down times.

 I think that the balancer has to be run manually in HDFS and
> there will be a maximum of 24 hrs window between a HDFS balancer execution
> and a major compaction during which data locality might be lost.
>

Yes the hdfs balancer needs to be run manually and yes it knows
nothing of how hbase has ordered the blocks and will not respect
region locality when it goes about its business.

I'm sure though I follow the rest of what you are saying above.

On locality, the fb lads are working on a primitive that makes it so
the hbase dfsclient will tell hdfs where to place blocks.  The favored
replica locations will be kept up in .META. in a new column When a
regionserver crashes, or if we want to move a region, we'll move it or
reopen it on one of the locations that has had region blocks
replicated to it.  This should help improve the locality story on
failover/move.

Without this functionality, we're left with the current behavior where
blocks for regions are scattered and its only per chance you'd have
good locality opening a region in any location other than the current
deploy where the gentle waves of compaction having been nudging data
local.

I dont believe there is an issue for the above yet.  Let me chase the
lads to file one.

This primitive that the lads are working on needs to be done I believe
before hbase-57 can be done (properly).  What you reckon Praveen?

St.Ack

Stack,

> Its recommended that you run major compactions yourself at down times.

Can we change the `hbase.hregion.majorcompaction` value from 86400000 to -1
along with the required code changes and make a note of it in the
hbase-default.xml? Also, the hbase.master.loadbalancer.class is not
specified in the hbase-default.xml. Should I open a JIRA and make those two
changes?

> In 0.92 there is hits per region and this gets reported to the master as
part of ClusterStatus as does memory usage.  This could be factored into a
new balance algorithm.

I just saw the code for the ClusterStatus, HServerLoad and RegionLoad.
First cut I was thinking if we can use only the resource usage (memory,
cpu, # of hits and not block location) to sort the region server in
decreasing order of resource usage and then shed the regions from the top
till the region server with an average usage is hit and then assign the
regions which have been shed to the region server list from the bottom.
This is more or less similar to the DefaultLoadBalancer, but taking the
resource usage into considerations and not on the # of regions.

Question is should memory, cpu and # of hits be given equal weightage or
make it configurable?

> On locality, the fb lads are working on a primitive that makes it so the
hbase dfsclient will tell hdfs where to place blocks.

Is there any JIRA for it?

So using this functionality HBase can specify the location of the 2nd and
the 3rd replica (1st is local) and later use the location of the 2nd and
3rd replica during a region server crash or a region movement. What happens
if the HDFS load balancer is run, the blocks are moved again?

> This primitive that the lads are working on needs to be done I believe
before hbase-57 can be done (properly).

Again the question is should we mix the memory, cpu, # of hits with the
block location and give them configurable weightage or keep the data
locality out for the first cut, since the major compaction (although manual
as per recommendation) will pull all the blocks together anyway.

Thanks,
Praveen

On Tue, Feb 21, 2012 at 2:39 AM, Stack <[EMAIL PROTECTED]> wrote:

> On Mon, Feb 20, 2012 at 5:03 AM, Praveen Sripati
> <[EMAIL PROTECTED]> wrote:
> > It would be nice to consider
> > both the resource usage of the region and the data locality into
> > consideration, not just purely based on the number of regions in the
> region
> > server as implemented currently.
> >
>
> Yes.
>
> > The file to block mapping can be found from the HDFS NameNode, but how to
> > find out which regions are loaded (# of requests, cpu and memory
> > perspective) and which are not? I could not see any resource utilization
> in
> > the region server pages.
> >
>
> In 0.92 there is hits per region and this gets reported to the master
> as part of ClusterStatus as does memory usage.  This could be factored
> into a new balance algorithm.  Could also send over cpu and hardware
> profile for factoring (though much of this is available via JMX --
> either we get these into clusterstatus or master does poll on jmx
> after it sees new server to get server profile)
>
> > Also, curious if HBASE-57 makes sense, since the major compaction runs
> > every 24 hrs
>
> Its recommended that you run major compactions yourself at down times.
>
>  I think that the balancer has to be run manually in HDFS and
> > there will be a maximum of 24 hrs window between a HDFS balancer
> execution
> > and a major compaction during which data locality might be lost.
> >
>
> Yes the hdfs balancer needs to be run manually and yes it knows
> nothing of how hbase has ordered the blocks and will not respect
> region locality when it goes about its business.
>
> I'm sure though I follow the rest of what you are saying above.
>
> On locality, the fb lads are working on a primitive that makes it so
> the hbase dfsclient will tell hdfs where to place blocks.  The favored
> replica locations will be kept up in .META. in a new column When a
> regionserver crashes, or if we want to move a region, we'll move it or

>>Its recommended that you run major compactions yourself at down times.
>
>Can we change the `hbase.hregion.majorcompaction` value from 86400000 to
>-1
>along with the required code changes and make a note of it in the
>hbase-default.xml? Also, the hbase.master.loadbalancer.class is not
>specified in the hbase-default.xml. Should I open a JIRA and make those
>two
>changes?

Note that the importance of non-peak major compactions should be mitigated
in 0.92 with multi-threaded compactions.  This config recommendation is
not a general solution but a way to cope with the existing
production-stable feature set.

On Mon, Feb 20, 2012 at 1:09 PM, Stack <[EMAIL PROTECTED]> wrote:
> On locality, the fb lads are working on a primitive that makes it so
> the hbase dfsclient will tell hdfs where to place blocks.

Here is the hdfs issue: https://issues.apache.org/jira/browse/HDFS-2576
St.Ack