-Re: Of hbase key distribution and query scalability, again.
Michael Segel 2012-05-26, 17:06
Could you show a use case where you see this happening?
If you have records that are being read that frequently, they would be cached in memory.
I think you could use some concept of a systems table and then using coprocessors you could update the table with the read patterns.
There'd be a performance hit. (I don't know how much of one but it will exist...) But its possible...
On May 26, 2012, at 11:45 AM, Ian Varley wrote:
> I gather that Dmitriy is asking whether there are any smarts in the region balancer based on heavy *read* traffic (i.e. if it turns out that your read load is heavily skewed towards a small subset of regions). Which there aren't, but could be if someone wanted to write the infrastructure for it (which would likely be complex, as you'd have to persist information about read traffic somewhere other than the logs). Then read-hot regions would be candidates for splitting, not just based on their size but also based on their read traffic.
> Caching is relevant to help read performance, for sure, but there could still be scenarios where your read traffic is all stuck in one region, and even after all other optimizations, it still leaves one region hot and the rest cold.
> To be totally clear, Dmitriy: I think this is a pretty advanced feature that's not high on the overall priority list, because in such a rare situation you could always manually split that region.
> On May 26, 2012, at 11:25 AM, Michael Segel wrote:
>> Jumping in on this late...
>>>>>> To cut a long story, is the region size the only current HBase
>>>>>> technique to balance load, esp. w.r.t query load? Or perhaps there are
>>>>>> some more advanced techniques to do that ?
>> So maybe I'm missing something but I don't see the problem.
>> In terms of writing data to be evenly/randomly distributed, you would hash the key (md5 or SHA-1 as examples).
>> This works well if you're doing get()s and not a lot of scan()s.
>> But on reads, how do you get 'hot spotting' ?
>> Should those rows be cached in memory?
>> So what am I missing? Besides another cup of coffee?
>> On May 25, 2012, at 1:23 PM, Ian Varley wrote:
>>> Yeah, I think you're right Dmitriy; there's nothing like that in HBase today as far as I know. If it'd be useful for you, maybe it would be for others, too; work up a rough patch and see what people think on the dev list.
>>> On May 25, 2012, at 1:02 PM, Dmitriy Lyubimov wrote:
>>>> Thanks, Ian.
>>>> I am talking about situation when even when we have uniform keys, the
>>>> query distribution over them is still non-uniform and impossible to
>>>> predict without sampling query skewness, but skewness is surprisingly
>>>> great. (as in least active/most active user may differ in activity 100
>>>> times and there is no way one could now which users are going to be
>>>> active and which are going to be not active). Assuming there are few
>>>> very active users, but many low active users, if two active users get
>>>> into the same region, it creates a hotspot which could have been
>>>> avoided if region balancer took notions of number of hits the regions
>>>> are getting recently.
>>>> Like i pointed out before, such skewness balancer could be fairly
>>>> easily implemented externally to hbase (as in TotalOrderPartitioner),
>>>> with exception that it would be interfering with the Hbase's balancer
>>>> itself so it must be integrated with the balancer in that case.
>>>> Also another distinct problem is time parameters of such balance
>>>> controller. The load may be changing fast enough or slow enough so
>>>> that sampling must be time-weighted itself.
>>>> All these tehchnicalities make it difficult to implement it outside
>>>> hbase or use key manipulation (as dynamic nature makes it difficult to
>>>> deal with key re-assigning to match newly discovered load