Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # user >> HBase split policy


Copy link to this message
-
Re: HBase split policy
Hi Jean

Before replying as to what i know, region splits can be configured too.

Ok, now on how the split happens
-> You can explicity ask the region to get splitted on a specific row key.
 If you know that splitting on that rowkey will yield you almost equal
region sizes.
-> Now when HBase tries to split, it just takes the midkey from the HFiles.
 Here the midkey is the one that is the first key in the mid block of the
HFile.
Also the individual rows cannot be split. So if one row is nearly the size
of the region and other rows are smaller in size, it tries to find the mid
block inside the HFile and the size of one the block is going to be very
huge and that may be splitted as one region.  I know this has to do with
the internals of the splitting code.
Regards
Ram

On Tue, Jan 22, 2013 at 5:12 PM, Jean-Marc Spaggiari <
[EMAIL PROTECTED]> wrote:

> Hi,
>
> I'm wondering, what is HBase split policy.
>
> I mean, let's imagine this situation.
>
> I have a region full of rows starting from AA to AZ. Thousands of
> hundreds. I also have few rows from B to DZ. Let's say only one
> hundred.
>
> Region is just above the maxfilesize, so it's fine.
>
> No, I add "A" and store a very big row into it. Almost half the size
> of my maxfilesize value. That mean it's now time to split this row.
>
> How will HBase decide where to split it? Is it going to use the
> lexical order? Which mean it will split somewhere between B and C? If
> it's done that way, I will have one VERY small region, and one VERY
> big which will still be over the maxfilesize and will need to be split
> again, and most probably many times, right?
>
> Or will HBase take the middle of the region, look at the closest key,
> and cut there?
>
> Yesterday, for one table, I merged all my regions into a single one.
> This gave me something like a 10GB region. Since I want to have at
> least 100 regions for this table, I have setup the maxfilesize to
> 100MB. I have restarted HBase, and let it worked over night.
>
> This morning, I have some very big regions, still over the 100MB, and
> some very small. And the big regions are at least hundred times bigger
> than the small one.
>
> I just stopped the cluster again to re-merge the regions into a single
> one and see if I have not done something wrong in the process, but in
> the meantime, I'm looking for more information about the way HBase is
> deciding where to cut, and if there is a way to customize that.
>
> Thanks,
>
> JM
>
> PS: Numbers are out of my head. I don't really recall how big the last
> region was yesterday. I will take more notes when the current
> MassMerge will be done.
>