Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # user >> Pre-split table using shell


Copy link to this message
-
Re: Pre-split table using shell
Ok...

Please tell me that this isn't a production system.

Is this on EC2?

On Jun 12, 2012, at 6:55 AM, Simon Kelly wrote:

> Thanks Michael
>
> I'm 100% sure its not the UUID distribution that's causing the problem. I'm
> going to try us the API to create the table and see if that changes things.
>
> The reason I want to pre-split the table is that HBase doesn't handle the
> initial load to a single regionserver and I can't start the system off
> slowly and allow a few splits to happen before fully loading it. Its 100%
> or nothing. I'm also stuck with only 8Gb of RAM per server and only 5
> servers so I need to try and get as much as I can from the get go.
>
> Simon
>
> On 12 June 2012 13:37, Michael Segel <[EMAIL PROTECTED]> wrote:
>
>> Ok,
>> Now that I'm awake, and am drinking my first cup of joe...
>>
>> If you just generate UUIDs you are not going to have an even distribution.
>> Nor are they going to be truly random due to how the machines are
>> generating their random numbers.
>> But this is not important in solving your problem....
>>
>> There is a set of UUIDs which are hashed and then truncated back down to a
>> 128 bit string.
>> You can generate the UUID, take a hash (SHA-1 or MD5) and then truncate it
>> to 128 bits.
>> This would generate a more random distribution across your splits.
>>
>> I'm also a bit curious about why you're pre-splitting in the first place.
>> I mean I understand why people do it, but its a short term fix and I
>> wonder how much pain you feel.
>>
>> Of course YMMV based on your use case.
>>
>> Hash your key and you'll be ok.
>>
>>
>>
>> On Jun 12, 2012, at 4:41 AM, Simon Kelly wrote:
>>
>>> Yes, I'm aware that UUID's are designed to be unique and not evenly
>>> distributed but I wouldn't expect a big gap in their distribution either.
>>>
>>> The other thing that is really confusing me is that the regions splits
>>> aren't lexicographical sorted. Perhaps there is a problem with the way
>> I'm
>>> specifying the splits in the split file. I haven't been able to find any
>>> docs on what format the splits keys should be in so I've used what's
>>> produced by Bytes.toStringBinary. Is that correct?
>>>
>>> Simon
>>>
>>> On 12 June 2012 10:23, Michael Segel <[EMAIL PROTECTED]> wrote:
>>>
>>>> UUIDs are unique but not necessarily random and even in random
>> samplings,
>>>> you may not see an even distribution except over time.
>>>>
>>>>
>>>> Sent from my iPhone
>>>>
>>>> On Jun 12, 2012, at 3:18 AM, "Simon Kelly" <[EMAIL PROTECTED]>
>> wrote:
>>>>
>>>>> Hi
>>>>>
>>>>> I'm getting some unexpected results with a pre-split table where some
>> of
>>>>> the regions are not getting any data.
>>>>>
>>>>> The table keys are UUID (generated using Java's UUID.randomUUID() )
>> which
>>>>> I'm storing as a byte[16]:
>>>>>
>>>>>  key[0-7] = uuid most significant bits
>>>>>  key[8-15] = uuid least significant bits
>>>>>
>>>>> The table is created via the shell as follows:
>>>>>
>>>>>  create 'table', {NAME => 'cf'}, {SPLITS_FILE => 'splits.txt'}
>>>>>
>>>>> The splits.txt is generated using the code here:
>>>>> http://pastebin.com/DAExXMDz which generates 32 regions split between
>>>> x00
>>>>> and xFF. I have also tried with 16 byte regions keys (x00x00... to
>>>>> xFFxFF...).
>>>>>
>>>>> As far as I understand this should distribute the rows evenly across
>> the
>>>>> regions but I'm getting a bunch of regions with no rows. I'm also
>>>> confused
>>>>> as the the ordering of the regions since it seems the start and end
>> keys
>>>>> aren't really matching up correctly. You can see the regions and the
>>>>> requests they are getting here: http://pastebin.com/B4771g5X
>>>>>
>>>>> Thanks in advance for the help.
>>>>> Simon
>>>>
>>
>>