Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase, mail # user - HBase Random Read latency > 100ms


Copy link to this message
-
Re: HBase Random Read latency > 100ms
lars hofhansl 2013-10-07, 21:04
He still should not see 100ms latency. 20ms, sure. 100ms seems large; there are still 8 machines serving the requests.

I agree this spec is far from optimal, but there is still something odd here.
Ramu, this does not look like a GC issue. You'd see much larger (worst case) latencies if that were the case (dozens of seconds).
Are you using 40 client from 40 different machines? Or from 40 different processes on the same machine? Or 40 threads in the same process?

Thanks.

-- Lars

________________________________
 From: Vladimir Rodionov <[EMAIL PROTECTED]>
To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>
Sent: Monday, October 7, 2013 11:02 AM
Subject: RE: HBase Random Read latency > 100ms
 

Ramu, your HBase configuration (128GB of heap) is far from optimal.
Nobody runs HBase with that amount of heap to my best knowledge.
32GB of RAM is the usual upper limit. We run 8-12GB in production.

What else, your IO capacity is VERY low. 2 SATA drives in RAID 1 for mostly random reads load?
You should have 8, better 12-16 drives per server. Forget about RAID. You have HDFS.

Block cache in your case does not help much , as since your read amplification is at least x20 (16KB block and 724 B read) - its just waste
RAM (heap). In your case you do not need LARGE heap and LARGE block cache.

I advise you reconsidering your hardware spec, applying all optimizations mentioned already in this thread and lowering your expectations.

With a right hardware you will be able to get 500-1000 truly random reads per server.

Best regards,
Vladimir Rodionov
Principal Platform Engineer
Carrier IQ, www.carrieriq.com
e-mail: [EMAIL PROTECTED]

________________________________________

From: Ramu M S [[EMAIL PROTECTED]]
Sent: Monday, October 07, 2013 5:23 AM
To: [EMAIL PROTECTED]
Subject: Re: HBase Random Read latency > 100ms

Hi Bharath,

I am little confused about the metrics displayed by Cloudera. Even when
there are no oeprations, the gc_time metric is showing 2s constant in the
graph. Is this the CMS gc_time (in that case no JVm pause) or the GC pause.

GC timings reported earlier is the average taken for gc_time metric for all
region servers.

Regards,
Ramu
On Mon, Oct 7, 2013 at 9:10 PM, Ramu M S <[EMAIL PROTECTED]> wrote:

> Jean,
>
> Yes. It is 2 drives.
>
> - Ramu
>
>
> On Mon, Oct 7, 2013 at 8:45 PM, Jean-Marc Spaggiari <
> [EMAIL PROTECTED]> wrote:
>
>> Quick questionon the disk side.
>>
>> When you say:
>> 800 GB SATA (7200 RPM) Disk
>> Is it 1x800GB? It's raid 1, so might be 2 drives? What's the
>> configuration?
>>
>> JM
>>
>>
>> 2013/10/7 Ramu M S <[EMAIL PROTECTED]>
>>
>> > Lars, Bharath,
>> >
>> > Compression is disabled for the table. This was not intended from the
>> > evaluation.
>> > I forgot to mention that during table creation. I will enable snappy
>> and do
>> > major compaction again.
>> >
>> > Please suggest other options to try out and also suggestions for the
>> > previous questions.
>> >
>> > Thanks,
>> > Ramu
>> >
>> >
>> > On Mon, Oct 7, 2013 at 6:35 PM, Ramu M S <[EMAIL PROTECTED]> wrote:
>> >
>> > > Bharath,
>> > >
>> > > I was about to report this. Yes indeed there is too much of GC time.
>> > > Just verified the GC time using Cloudera Manager statistics(Every
>> minute
>> > > update).
>> > >
>> > > For each Region Server,
>> > >  - During Read: Graph shows 2s constant.
>> > >  - During Compaction: Graph starts with 7s and goes as high as 20s
>> during
>> > > end.
>> > >
>> > > Few more questions,
>> > > 1. For the current evaluation, since the reads are completely random
>> and
>> > I
>> > > don't expect to read same data again can I set the Heap to the
>> default 1
>> > GB
>> > > ?
>> > >
>> > > 2. Can I completely turn off BLOCK CACHE for this table?
>> > >    http://hbase.apache.org/book/regionserver.arch.html recommends
>> that
>> > > for Randm reads.
>> > >
>> > > 3. But in the next phase of evaluation, We are interested to use
>> HBase as
>> > > In-memory KV DB by having the latest data in RAM (To the tune of

Confidentiality Notice:  The information contained in this message, including any attachments hereto, may be confidential and is intended to be read only by the individual or entity to whom this message is addressed. If the reader of this message is not the intended recipient or an agent or designee of the intended recipient, please note that any review, use, disclosure or distribution of this message or its attachments, in any form, is strictly prohibited.  If you have received this message in error, please immediately notify the sender and/or [EMAIL PROTECTED] and delete or destroy any copy of this message and its attachments.