Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # user >> Improving HBase read performance (based on YCSB)


Copy link to this message
-
Re: Improving HBase read performance (based on YCSB)
Yep, definitely bound on seeks - see the 100% util, and the r/s >100.
The bandwidth provided by random IO from a disk is going to be much
smaller than the sequential IO you see from hdparm

-Todd

On Tue, Feb 14, 2012 at 3:06 PM, Bharath Ravi <[EMAIL PROTECTED]> wrote:
> Thanks Todd!
>
> I check disk bandwidth by first running "hparm" on it, (this shows me a
> read b/w of around 56Mbps)
> and then running "iftop" while the benchmarks run (This shows me that reads
> are only around 10-15Mbps: but
> this could definitely be because random seeks are a bottleneck)
>
> The iostat output seems to suggest seek is a problem too, although I'm not
> sure I interpret these numbers correctly.
> Here's some output from iostat, while the benchmark runs:
>
> Do the queue-lengths I see here indicate a bottleneck?
>
> Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s avgrq-sz
> avgqu-sz   await r_await w_await  svctm  %util
> sda               1.00     1.28    6.31    2.57     0.43     1.01   332.30
>    3.63  408.90  259.25  776.48   6.40   5.67
> dm-0              0.00     0.00    6.55    2.87     0.43     1.01   311.48
>    4.60  487.90  380.14  733.49   5.99   5.65
> dm-1              0.00     0.00    0.29    0.88     0.00     0.00     8.00
>    1.33 1135.17   89.17 1479.15   3.23   0.38
>
> Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s avgrq-sz
> avgqu-sz   await r_await w_await  svctm  %util
> sda               1.40    18.20  257.80    2.60    13.84     0.08   109.49
>   62.55  240.45  241.44  141.85   4.43 115.28
> dm-0              0.00     0.00  258.80    3.40    13.81     0.01   107.99
>   63.17  241.17  241.93  183.76   4.40 115.28
> dm-1              0.00     0.00    0.00   17.20     0.00     0.07     8.00
>    0.21   12.00    0.00   12.00   0.14   0.24
>
> Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s avgrq-sz
> avgqu-sz   await r_await w_await  svctm  %util
> sda               1.60   255.80  234.60    3.00    16.43     1.01   150.36
>  113.11  427.74  430.83  186.40   4.82 114.56
> dm-0              0.00     0.00  262.00    1.00    18.03     0.00   140.44
>  113.86  389.06  389.87  175.20   4.36 114.56
> dm-1              0.00     0.00    0.20  258.00     0.00     1.01     8.00
>   37.92  146.87    0.00  146.98   1.02  26.32
>
> Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s avgrq-sz
> avgqu-sz   await r_await w_await  svctm  %util
> sda               3.42    98.67  152.09    2.66     9.47     0.40   130.68
>   75.82  543.73  549.56  210.86   4.61  71.33
> dm-0              0.00     0.00  132.32    1.33     8.04     0.01   123.43
>   76.06  631.83  635.09  308.00   5.34  71.33
> dm-1              0.00     0.00    3.04   99.62     0.01     0.39     8.00
>   14.84  144.57  648.75  129.18   2.72  27.91
>
> Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s avgrq-sz
> avgqu-sz   await r_await w_await  svctm  %util
> sda               1.27    54.85  220.89    2.95    13.15     0.23   122.44
>   66.15  304.77  305.75  231.71   4.43  99.16
> dm-0              0.00     0.00  232.49    3.38    14.07     0.02   122.30
>   66.66  291.36  292.25  230.00   4.20  99.16
> dm-1              0.00     0.00    0.00   54.22     0.00     0.21     8.00
>   18.12  334.27    0.00  334.27   1.57   8.52
>
> Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s avgrq-sz
> avgqu-sz   await r_await w_await  svctm  %util
> sda               4.00    19.80  224.60    3.00    12.80     0.09   115.95
>   62.69  254.17  256.84   54.67   4.39  99.92
> dm-0              0.00     0.00  229.40    2.20    13.00     0.01   115.01
>   61.95  246.34  247.99   73.82   4.29  99.28
> dm-1              0.00     0.00    8.00   20.40     0.03     0.08     8.00
>    3.78  133.13  216.40  100.47  14.25  40.48
>
> Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s avgrq-sz
> avgqu-sz   await r_await w_await  svctm  %util
> sda               0.59   467.12  191.98    5.28    11.99     1.84   143.64

Todd Lipcon
Software Engineer, Cloudera
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB