Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # user >> Improving HBase read performance (based on YCSB)


Copy link to this message
-
Re: Improving HBase read performance (based on YCSB)
Yep, definitely bound on seeks - see the 100% util, and the r/s >100.
The bandwidth provided by random IO from a disk is going to be much
smaller than the sequential IO you see from hdparm

-Todd

On Tue, Feb 14, 2012 at 3:06 PM, Bharath Ravi <[EMAIL PROTECTED]> wrote:
> Thanks Todd!
>
> I check disk bandwidth by first running "hparm" on it, (this shows me a
> read b/w of around 56Mbps)
> and then running "iftop" while the benchmarks run (This shows me that reads
> are only around 10-15Mbps: but
> this could definitely be because random seeks are a bottleneck)
>
> The iostat output seems to suggest seek is a problem too, although I'm not
> sure I interpret these numbers correctly.
> Here's some output from iostat, while the benchmark runs:
>
> Do the queue-lengths I see here indicate a bottleneck?
>
> Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s avgrq-sz
> avgqu-sz   await r_await w_await  svctm  %util
> sda               1.00     1.28    6.31    2.57     0.43     1.01   332.30
>    3.63  408.90  259.25  776.48   6.40   5.67
> dm-0              0.00     0.00    6.55    2.87     0.43     1.01   311.48
>    4.60  487.90  380.14  733.49   5.99   5.65
> dm-1              0.00     0.00    0.29    0.88     0.00     0.00     8.00
>    1.33 1135.17   89.17 1479.15   3.23   0.38
>
> Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s avgrq-sz
> avgqu-sz   await r_await w_await  svctm  %util
> sda               1.40    18.20  257.80    2.60    13.84     0.08   109.49
>   62.55  240.45  241.44  141.85   4.43 115.28
> dm-0              0.00     0.00  258.80    3.40    13.81     0.01   107.99
>   63.17  241.17  241.93  183.76   4.40 115.28
> dm-1              0.00     0.00    0.00   17.20     0.00     0.07     8.00
>    0.21   12.00    0.00   12.00   0.14   0.24
>
> Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s avgrq-sz
> avgqu-sz   await r_await w_await  svctm  %util
> sda               1.60   255.80  234.60    3.00    16.43     1.01   150.36
>  113.11  427.74  430.83  186.40   4.82 114.56
> dm-0              0.00     0.00  262.00    1.00    18.03     0.00   140.44
>  113.86  389.06  389.87  175.20   4.36 114.56
> dm-1              0.00     0.00    0.20  258.00     0.00     1.01     8.00
>   37.92  146.87    0.00  146.98   1.02  26.32
>
> Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s avgrq-sz
> avgqu-sz   await r_await w_await  svctm  %util
> sda               3.42    98.67  152.09    2.66     9.47     0.40   130.68
>   75.82  543.73  549.56  210.86   4.61  71.33
> dm-0              0.00     0.00  132.32    1.33     8.04     0.01   123.43
>   76.06  631.83  635.09  308.00   5.34  71.33
> dm-1              0.00     0.00    3.04   99.62     0.01     0.39     8.00
>   14.84  144.57  648.75  129.18   2.72  27.91
>
> Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s avgrq-sz
> avgqu-sz   await r_await w_await  svctm  %util
> sda               1.27    54.85  220.89    2.95    13.15     0.23   122.44
>   66.15  304.77  305.75  231.71   4.43  99.16
> dm-0              0.00     0.00  232.49    3.38    14.07     0.02   122.30
>   66.66  291.36  292.25  230.00   4.20  99.16
> dm-1              0.00     0.00    0.00   54.22     0.00     0.21     8.00
>   18.12  334.27    0.00  334.27   1.57   8.52
>
> Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s avgrq-sz
> avgqu-sz   await r_await w_await  svctm  %util
> sda               4.00    19.80  224.60    3.00    12.80     0.09   115.95
>   62.69  254.17  256.84   54.67   4.39  99.92
> dm-0              0.00     0.00  229.40    2.20    13.00     0.01   115.01
>   61.95  246.34  247.99   73.82   4.29  99.28
> dm-1              0.00     0.00    8.00   20.40     0.03     0.08     8.00
>    3.78  133.13  216.40  100.47  14.25  40.48
>
> Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s avgrq-sz
> avgqu-sz   await r_await w_await  svctm  %util
> sda               0.59   467.12  191.98    5.28    11.99     1.84   143.64

Todd Lipcon
Software Engineer, Cloudera