Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # user >> HBase Random Read latency > 100ms


Copy link to this message
-
Re: HBase Random Read latency > 100ms
Hi All,

I just ran only 8 parallel clients,

With SCR Enabled : Average Latency is 80 ms, IO Wait % is around 8
With SCR Disabled: Average Latency is 40 ms, IO Wait % is around 2

I always thought SCR enabled, allows a client co-located with the DataNode
to read HDFS file blocks directly. This gives a performance boost to
distributed clients that are aware of locality.

Is my understanding wrong OR it doesn't apply to my scenario?

Meanwhile I will try setting the parameter suggested by Lars and post you
the results.

Thanks,
Ramu
On Wed, Oct 9, 2013 at 2:29 PM, lars hofhansl <[EMAIL PROTECTED]> wrote:

> Good call.
> Could try to enable hbase.regionserver.checksum.verify, which will cause
> HBase to do its own checksums rather than relying on HDFS (and which saves
> 1 IO per block get).
>
> I do think you can expect the index blocks to be cached at all times.
>
> -- Lars
> ________________________________
> From: Vladimir Rodionov <[EMAIL PROTECTED]>
> To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>
> Sent: Tuesday, October 8, 2013 8:44 PM
> Subject: RE: HBase Random Read latency > 100ms
>
>
> Upd.
>
> Each HBase Get = 2 HDFS read IO (index block + data block)= 4 File IO
> (data + .crc) in a worst case. I think if Bloom Filter is enabled than
> it is going to be 6 File IO in a worst case (large data set), therefore
> you will have not 5 IO requests in queue but up to 20-30 IO requests in a
> queue
> This definitely explains > 100ms avg latency.
>
>
>
> Best regards,
> Vladimir Rodionov
> Principal Platform Engineer
> Carrier IQ, www.carrieriq.com
> e-mail: [EMAIL PROTECTED]
>
> ________________________________________
>
> From: Vladimir Rodionov
> Sent: Tuesday, October 08, 2013 7:24 PM
> To: [EMAIL PROTECTED]
> Subject: RE: HBase Random Read latency > 100ms
>
> Ramu,
>
> You have 8 server boxes and 10 client. You have 40 requests in parallel -
> 5 per RS/DN?
>
> You have 5 requests on random reads in a IO queue of your single RAID1.
> With avg read latency of 10 ms, 5 requests in queue will give us 30ms. Add
> some overhead
> of HDFS + HBase and you will probably have your issue explained ?
>
> Your bottleneck is your disk system, I think. When you serve most of
> requests from disks as in your large data set scenario, make sure you have
> adequate disk sub-system and
> that it is configured properly. Block Cache and OS page can not help you
> in this case as working data set is larger than both caches.
>
> Good performance numbers in small data set scenario are explained by the
> fact that data fits into OS page cache and Block Cache - you do not read
> data from disk even if
> you disable block cache.
>
>
> Best regards,
> Vladimir Rodionov
> Principal Platform Engineer
> Carrier IQ, www.carrieriq.com
> e-mail: [EMAIL PROTECTED]
>
> ________________________________________
> From: Ramu M S [[EMAIL PROTECTED]]
> Sent: Tuesday, October 08, 2013 6:00 PM
> To: [EMAIL PROTECTED]
> Subject: Re: HBase Random Read latency > 100ms
>
> Hi All,
>
> After few suggestions from the mails earlier I changed the following,
>
> 1. Heap Size to 16 GB
> 2. Block Size to 16KB
> 3. HFile size to 8 GB (Table now has 256 regions, 32 per server)
> 4. Data Locality Index is 100 in all RS
>
> I have clients running in 10 machines, each with 4 threads. So total 40.
> This is same in all tests.
>
> Result:
>            1. Average latency is still >100ms.
>            2. Heap occupancy is around 2-2.5 GB in all RS
>
> Few more tests carried out yesterday,
>
> TEST 1: Small data set (100 Million records, each with 724 bytes).
> ==========================================> Configurations:
> 1. Heap Size to 1 GB
> 2. Block Size to 16KB
> 3. HFile size to 1 GB (Table now has 128 regions, 16 per server)
> 4. Data Locality Index is 100 in all RS
>
> I disabled Block Cache on the table, to make sure I read everything from
> disk, most of the time.
>
> Result:
>    1. Average Latency is 8ms and throughput went up to 6K/Sec per RS.
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB