Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # user >> HBase Random Read latency > 100ms


Copy link to this message
-
Re: HBase Random Read latency > 100ms
What is the iowait in both cases ?
 
Regards,
- kiru
Kiru Pakkirisamy | webcloudtech.wordpress.com
________________________________
 From: Ramu M S <[EMAIL PROTECTED]>
To: [EMAIL PROTECTED]
Sent: Tuesday, October 8, 2013 6:00 PM
Subject: Re: HBase Random Read latency > 100ms
 

Hi All,

After few suggestions from the mails earlier I changed the following,

1. Heap Size to 16 GB
2. Block Size to 16KB
3. HFile size to 8 GB (Table now has 256 regions, 32 per server)
4. Data Locality Index is 100 in all RS

I have clients running in 10 machines, each with 4 threads. So total 40.
This is same in all tests.

Result:
           1. Average latency is still >100ms.
           2. Heap occupancy is around 2-2.5 GB in all RS

Few more tests carried out yesterday,

TEST 1: Small data set (100 Million records, each with 724 bytes).
==========================================Configurations:
1. Heap Size to 1 GB
2. Block Size to 16KB
3. HFile size to 1 GB (Table now has 128 regions, 16 per server)
4. Data Locality Index is 100 in all RS

I disabled Block Cache on the table, to make sure I read everything from
disk, most of the time.

Result:
   1. Average Latency is 8ms and throughput went up to 6K/Sec per RS.
   2. With Block Cache enabled again, I got average latency around 2ms
and throughput of 10K/Sec per RS.
       Heap occupancy around 650 MB
   3. Increased the Heap to 16GB, with Block Cache still enabled, I got
average latency around 1 ms and throughput 20K/Sec per RS
       Heap Occupancy around 2-2.5 GB in all RS

TEST 2: Large Data set (1.8 Billion records, each with 724 bytes)
=================================================Configurations:
1. Heap Size to 1 GB
2. Block Size to 16KB
3. HFile size to 1 GB (Table now has 2048 regions, 256 per server)
4. Data Locality Index is 100 in all RS

Result:
  1. Average Latency is > 500ms to start with and gradually decreases, but
even after around 100 Million reads it is still >100 ms
  2. Block Cache = TRUE/FALSE does not make any difference here. Even Heap
Size (1GB / 16GB) does not make any difference.
  3. Heap occupancy is around 2-2.5 GB under 16GB Heap and around 650 MB
under 1GB Heap.

GC Time in all of the scenarios is around 2ms/Second, as shown in the
Cloudera Manager.

Reading most of the items from Disk in less data scenario gives better
results and very low latencies.

Number of regions per RS and HFile size does make a huge difference in my
Cluster.
Keeping 100 Regions per RS as max(Most of the discussions suggest this),
should I restrict the HFile size to 1GB? and thus reducing the storage
capacity (From 700 GB to 100GB per RS)?

Please advice.

Thanks,
Ramu
On Wed, Oct 9, 2013 at 4:58 AM, Vladimir Rodionov
<[EMAIL PROTECTED]>wrote:

> What are your current heap and block cache sizes?
>
> Best regards,
> Vladimir Rodionov
> Principal Platform Engineer
> Carrier IQ, www.carrieriq.com
> e-mail: [EMAIL PROTECTED]
>
> ________________________________________
> From: Ramu M S [[EMAIL PROTECTED]]
> Sent: Monday, October 07, 2013 10:55 PM
> To: [EMAIL PROTECTED]
> Subject: Re: HBase Random Read latency > 100ms
>
> Hi All,
>
> Average Latency is still around 80ms.
> I have done the following,
>
> 1. Enabled Snappy Compression
> 2. Reduce the HFile size to 8 GB
>
> Should I attribute these results to bad Disk Configuration OR anything else
> to investigate?
>
> - Ramu
>
>
> On Tue, Oct 8, 2013 at 10:56 AM, Ramu M S <[EMAIL PROTECTED]> wrote:
>
> > Vladimir,
> >
> > Thanks for the Insights into Future Caching features. Looks very
> > interesting.
> >
> > - Ramu
> >
> >
> > On Tue, Oct 8, 2013 at 10:45 AM, Vladimir Rodionov <
> > [EMAIL PROTECTED]> wrote:
> >
> >> Ramu,
> >>
> >> If your working set of data fits into 192GB you may get additional boost
> >> by utilizing OS page cache, or wait until
> >> 0.98 release which introduces new bucket cache implementation (port of
> >> Facebook L2 cache). You can try vanilla bucket cache in 0.96 (not
> released
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB