Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # user >> Pre-split table using shell


Copy link to this message
-
Re: Pre-split table using shell
Ok,
Now that I'm awake, and am drinking my first cup of joe...

If you just generate UUIDs you are not going to have an even distribution.
Nor are they going to be truly random due to how the machines are generating their random numbers.
But this is not important in solving your problem....

There is a set of UUIDs which are hashed and then truncated back down to a 128 bit string.
You can generate the UUID, take a hash (SHA-1 or MD5) and then truncate it to 128 bits.
This would generate a more random distribution across your splits.

I'm also a bit curious about why you're pre-splitting in the first place.
I mean I understand why people do it, but its a short term fix and I wonder how much pain you feel.

Of course YMMV based on your use case.

Hash your key and you'll be ok.

On Jun 12, 2012, at 4:41 AM, Simon Kelly wrote:

> Yes, I'm aware that UUID's are designed to be unique and not evenly
> distributed but I wouldn't expect a big gap in their distribution either.
>
> The other thing that is really confusing me is that the regions splits
> aren't lexicographical sorted. Perhaps there is a problem with the way I'm
> specifying the splits in the split file. I haven't been able to find any
> docs on what format the splits keys should be in so I've used what's
> produced by Bytes.toStringBinary. Is that correct?
>
> Simon
>
> On 12 June 2012 10:23, Michael Segel <[EMAIL PROTECTED]> wrote:
>
>> UUIDs are unique but not necessarily random and even in random samplings,
>> you may not see an even distribution except over time.
>>
>>
>> Sent from my iPhone
>>
>> On Jun 12, 2012, at 3:18 AM, "Simon Kelly" <[EMAIL PROTECTED]> wrote:
>>
>>> Hi
>>>
>>> I'm getting some unexpected results with a pre-split table where some of
>>> the regions are not getting any data.
>>>
>>> The table keys are UUID (generated using Java's UUID.randomUUID() ) which
>>> I'm storing as a byte[16]:
>>>
>>>   key[0-7] = uuid most significant bits
>>>   key[8-15] = uuid least significant bits
>>>
>>> The table is created via the shell as follows:
>>>
>>>   create 'table', {NAME => 'cf'}, {SPLITS_FILE => 'splits.txt'}
>>>
>>> The splits.txt is generated using the code here:
>>> http://pastebin.com/DAExXMDz which generates 32 regions split between
>> x00
>>> and xFF. I have also tried with 16 byte regions keys (x00x00... to
>>> xFFxFF...).
>>>
>>> As far as I understand this should distribute the rows evenly across the
>>> regions but I'm getting a bunch of regions with no rows. I'm also
>> confused
>>> as the the ordering of the regions since it seems the start and end keys
>>> aren't really matching up correctly. You can see the regions and the
>>> requests they are getting here: http://pastebin.com/B4771g5X
>>>
>>> Thanks in advance for the help.
>>> Simon
>>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB