Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # user >> Pre-split table using shell

Copy link to this message
Re: Pre-split table using shell

Please tell me that this isn't a production system.

Is this on EC2?

On Jun 12, 2012, at 6:55 AM, Simon Kelly wrote:

> Thanks Michael
> I'm 100% sure its not the UUID distribution that's causing the problem. I'm
> going to try us the API to create the table and see if that changes things.
> The reason I want to pre-split the table is that HBase doesn't handle the
> initial load to a single regionserver and I can't start the system off
> slowly and allow a few splits to happen before fully loading it. Its 100%
> or nothing. I'm also stuck with only 8Gb of RAM per server and only 5
> servers so I need to try and get as much as I can from the get go.
> Simon
> On 12 June 2012 13:37, Michael Segel <[EMAIL PROTECTED]> wrote:
>> Ok,
>> Now that I'm awake, and am drinking my first cup of joe...
>> If you just generate UUIDs you are not going to have an even distribution.
>> Nor are they going to be truly random due to how the machines are
>> generating their random numbers.
>> But this is not important in solving your problem....
>> There is a set of UUIDs which are hashed and then truncated back down to a
>> 128 bit string.
>> You can generate the UUID, take a hash (SHA-1 or MD5) and then truncate it
>> to 128 bits.
>> This would generate a more random distribution across your splits.
>> I'm also a bit curious about why you're pre-splitting in the first place.
>> I mean I understand why people do it, but its a short term fix and I
>> wonder how much pain you feel.
>> Of course YMMV based on your use case.
>> Hash your key and you'll be ok.
>> On Jun 12, 2012, at 4:41 AM, Simon Kelly wrote:
>>> Yes, I'm aware that UUID's are designed to be unique and not evenly
>>> distributed but I wouldn't expect a big gap in their distribution either.
>>> The other thing that is really confusing me is that the regions splits
>>> aren't lexicographical sorted. Perhaps there is a problem with the way
>> I'm
>>> specifying the splits in the split file. I haven't been able to find any
>>> docs on what format the splits keys should be in so I've used what's
>>> produced by Bytes.toStringBinary. Is that correct?
>>> Simon
>>> On 12 June 2012 10:23, Michael Segel <[EMAIL PROTECTED]> wrote:
>>>> UUIDs are unique but not necessarily random and even in random
>> samplings,
>>>> you may not see an even distribution except over time.
>>>> Sent from my iPhone
>>>> On Jun 12, 2012, at 3:18 AM, "Simon Kelly" <[EMAIL PROTECTED]>
>> wrote:
>>>>> Hi
>>>>> I'm getting some unexpected results with a pre-split table where some
>> of
>>>>> the regions are not getting any data.
>>>>> The table keys are UUID (generated using Java's UUID.randomUUID() )
>> which
>>>>> I'm storing as a byte[16]:
>>>>>  key[0-7] = uuid most significant bits
>>>>>  key[8-15] = uuid least significant bits
>>>>> The table is created via the shell as follows:
>>>>>  create 'table', {NAME => 'cf'}, {SPLITS_FILE => 'splits.txt'}
>>>>> The splits.txt is generated using the code here:
>>>>> http://pastebin.com/DAExXMDz which generates 32 regions split between
>>>> x00
>>>>> and xFF. I have also tried with 16 byte regions keys (x00x00... to
>>>>> xFFxFF...).
>>>>> As far as I understand this should distribute the rows evenly across
>> the
>>>>> regions but I'm getting a bunch of regions with no rows. I'm also
>>>> confused
>>>>> as the the ordering of the regions since it seems the start and end
>> keys
>>>>> aren't really matching up correctly. You can see the regions and the
>>>>> requests they are getting here: http://pastebin.com/B4771g5X
>>>>> Thanks in advance for the help.
>>>>> Simon