Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
HBase >> mail # user >> experiencing high latency for few reads in HBase


+
Saurabh Yahoo 2013-08-28, 12:10
+
Vladimir Rodionov 2013-08-28, 19:15
+
Saurabh Yahoo 2013-08-28, 22:17
+
lars hofhansl 2013-08-29, 04:29
+
Federico Gaule 2013-08-29, 12:37
+
Saurabh Yahoo 2013-08-29, 17:47
+
Kiru Pakkirisamy 2013-08-30, 02:45
+
Adrien Mogenet 2013-08-30, 06:11
+
Kiru Pakkirisamy 2013-08-30, 08:36
+
Kiru Pakkirisamy 2013-08-28, 20:30
+
Saurabh Yahoo 2013-08-28, 22:20
+
Kiru Pakkirisamy 2013-08-28, 23:01
+
Saurabh Yahoo 2013-08-28, 23:18
+
Vladimir Rodionov 2013-08-29, 00:10
+
Saurabh Yahoo 2013-08-29, 09:49
Copy link to this message
-
Re: experiencing high latency for few reads in HBase
Saurabh,
I have a suspicion that the few high latency responses are happening because of "hot" region.(s)
I vaguely remember you mentioning that the data is evenly distributed across all regions.
I hope your test also goes across them evenly. You may want to check the read requests to the regions.
 
Regards,
- kiru
________________________________
 From: Saurabh Yahoo <[EMAIL PROTECTED]>
To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>
Cc: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>
Sent: Thursday, August 29, 2013 2:49 AM
Subject: Re: experiencing high latency for few reads in HBase
 

Hi Vlad,

We do have strict latency requirement as it is financial data requiring direct access from clients.

Are you saying that it is not possible to achieve sub second latency using hbase (because it is based on java.) ?

On Aug 28, 2013, at 8:10 PM, Vladimir Rodionov <[EMAIL PROTECTED]> wrote:

> Increasing Java heap size will make latency worse, actually.
> You can't guarantee 1 sec max latency if run Java app (unless your heap size is much less than 1GB).
> I have never heard about strict maximum latency limit. Usually , its 99% , 99.9 or 99.99% query percentiles.
>
> You can greatly reduce your 99.xxx% percentile latency by storing you data in 2 replicas to two different region servers.
> Issue two read operations to those two region servers in parallel and get the first response. Probability theory states that  probability
> of two independent events (slow requests) is  the product of event's probabilities themselves.
>
>
> Best regards,
> Vladimir Rodionov
> Principal Platform Engineer
> Carrier IQ, www.carrieriq.com
> e-mail: [EMAIL PROTECTED]
>
> ________________________________________
> From: Saurabh Yahoo [[EMAIL PROTECTED]]
> Sent: Wednesday, August 28, 2013 4:18 PM
> To: [EMAIL PROTECTED]
> Subject: Re: experiencing high latency for few reads in HBase
>
> Thanks Kiru,
>
> Scan is not an option for our use cases.  Our read is pretty random.
>
> Any other suggestion to bring down the latency.
>
> Thanks,
> Saurabh.
>
>
> On Aug 28, 2013, at 7:01 PM, Kiru Pakkirisamy <[EMAIL PROTECTED]> wrote:
>
>> Saurabh, we are able to 600K rowxcolumns in 400 msec. We have put what was a 40million row table as 400K rows and columns. We Get about 100 of the rows from this 400K , do quite a bit of calculations in the coprocessor (almost a group-order by) and return in this time.
>> Maybe should consider replacing the MultiGets with Scan with Filter. I like the FuzzyRowFilter even though you might need to match with exact key. It works only with fixed length key.
>> (I do have an issue right now, it is not scaling to multiple clients.)
>>
>> Regards,
>> - kiru
>>
>>
>> Kiru Pakkirisamy | webcloudtech.wordpress.com
>>
>>
>> ________________________________
>> From: Saurabh Yahoo <[EMAIL PROTECTED]>
>> To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>
>> Cc: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>
>> Sent: Wednesday, August 28, 2013 3:20 PM
>> Subject: Re: experiencing high latency for few reads in HBase
>>
>>
>> Thanks Kitu. We need less than 1 sec latency.
>>
>> We are using both muliGet and get.
>>
>> We have three concurrent clients running 10 threads each. ( that makes total 30 concurrent clients).
>>
>> Thanks,
>> Saurabh.
>>
>> On Aug 28, 2013, at 4:30 PM, Kiru Pakkirisamy <[EMAIL PROTECTED]> wrote:
>>
>>> Right 4 sec is good.
>>> @Saurabh - so your read is - getting 20 out of 25 millions rows ?. Is this a Get or a Scan ?
>>> BTW, in this stress test how many concurrent clients do you have ?
>>>
>>> Regards,
>>> - kiru
>>>
>>>
>>> ________________________________
>>> From: Vladimir Rodionov <[EMAIL PROTECTED]>
>>> To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>
>>> Sent: Wednesday, August 28, 2013 12:15 PM
>>> Subject: RE: experiencing high latency for few reads in HBase
>>>
>>>
>>> 1. 4 sec max latency is not that bad taking into account 12GB heap.  It can be much larger. What is your SLA?
+
Saurabh Yahoo 2013-08-29, 17:49
+
Vladimir Rodionov 2013-08-29, 16:17
+
Saurabh Yahoo 2013-08-29, 17:52
+
Bryan Beaudreault 2013-08-29, 17:55
+
Vladimir Rodionov 2013-08-29, 18:11
+
Kiru Pakkirisamy 2013-08-29, 18:33
+
Saurabh Yahoo 2013-08-29, 19:09
+
Kiru Pakkirisamy 2013-08-29, 19:27
+
Adrien Mogenet 2013-08-29, 20:07
+
Saurabh Yahoo 2013-08-29, 20:58
+
Vladimir Rodionov 2013-08-28, 19:18
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB