-Re: scaling a low latency service with HBase
Dave Latham 2012-10-22, 23:37
On Fri, Oct 19, 2012 at 5:22 PM, Amandeep Khurana <[EMAIL PROTECTED]> wrote:
> Answers inline
> On Fri, Oct 19, 2012 at 4:31 PM, Dave Latham <[EMAIL PROTECTED]> wrote:
>> I need to scale an internal service / datastore that is currently hosted on
>> an HBase cluster and wanted to ask for advice from anyone out there who may
>> have some to share. The service does simple key value lookups on 20 byte
>> keys to 20-40 byte values. It currently has about 5 billion entries
>> (200GB), and processes about 40k random reads per second, and about 2k
>> random writes per second. It currently delivers a median response at 2ms,
>> 90% at 20ms, 99% at 200ms, 99.5% at 5000ms - but the mean is 58ms which is
>> no longer meeting our needs very well. It is persistent and highly
>> available. I need to measure its working set more closely, but I believe
>> that around 20-30% (randomly distributed) of the data is accessed each
>> day. I want a system that can scale to at least 10x current levels (50
>> billion entries - 2TB, 400k requests per second) and achieve a mean < 5ms
>> (ideally 1-2ms) and 99.5% < 50ms response time for reads while maintaining
>> persistence and reasonably high availability (99.9%). Writes would ideally
>> be in the same as range but we could probably tolerate a mean more in the
>> 20-30ms range.
>> Clearly for that latency, spinning disks won't cut it. The current service
>> is running out of an hbase cluster that is shared with many other things
>> and when those other things hit the disk and network hard is when it
>> degrades. The cluster has hundreds of nodes and this data is fits in a
>> small slice of block cache across most of them. The concerns are that its
>> performance is impacted by other loads and that as it continues to grow
>> there may not be enough space in the current cluster's shared block cache.
>> So I'm looking for something that will serve out of memory (backed by disk
>> for persistence) or from SSDs. A few questions that I would love to hear
>> answers for:
>> - Does HBase sound like a good match as this grows?
> Yes. The key to get more predictable performance is to separate out
> workloads. What are the other things that are using the same physical
> hardware and impacting performance? Have you measure performance when
> nothing else is running on the cluster?
There are several other things sharing the cluster and using it more
heavily than this service - both online request handling as well as
some large batch map reduce jobs. When the large jobs aren't running
the performance is acceptable and typically in the 1-2ms mean reads
range. (Served out of block cache).
>> - Does anyone have experience running HBase over SSDs? What sort of
>> latency and requests per second have you been able to achieve?
> I don't believe many people are actually running this in production yet.
> Some folks have done some research on this topic and posted blogs (eg:
> but there's not a whole lot more than that to go by at this point.
Thanks, that's a really helpful reference. It sounds like it could be
a big gain over disks already but that the bottleneck would move from
IO to CPU and that there would be significant work to be done.
>> - Is anyone using a row cache on top of (or built into) HBase? I think
>> there's been a bit of discussion on occasion but it hasn't gone very far.
>> There would be some overhead for each row. It seems that if we were to
>> continue to rely on memory + disks this could reduce the memory required.
>> - Does anyone have alternate suggestions for such a service?
> The biggest recommendation is to separate out the workloads and then start
> planning for more hardware or additional components to get better
Right, that's why I'm looking to separate this service out. However,
I'd like to go with a much smaller set of nodes for this particular
service rather than duplicating the large, expensive cluster.