Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce, mail # user - I/O stats interpretation during concurrent hive M/R runs


Copy link to this message
-
Re: I/O stats interpretation during concurrent hive M/R runs
Himanish Kushary 2012-08-23, 19:44
After sending this message I issued the iostat -dxm 5 command on the
DNs.... the %util column shows 70-80  average value sometimes going up to
90-100 for few seconds

Does this mean the disk is becoming the bottleneck ? or is this normal ?

On Thu, Aug 23, 2012 at 3:14 PM, Himanish Kushary <[EMAIL PROTECTED]>wrote:

> Hi,
>
> I am curious about interpretation of the output from iostat on a datanode
> during a M/R run.I want to understand how to diagnosis a disk i/o issue in
> a hadoop cluster.
>
> Is there any good documentation to help me understand the results from
> iostats in Hadoop context ?
>
> Here are the iostat output  from a DN while two intensive M/R jobs were
> executing.Does this result indicate any performance issue related to the
> disks ?
>
> Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
> sda               1.80         0.00        59.20          0        296
> sdb            1436.20     96376.00    211424.00     481880    1057120
>
> avg-cpu:  %user   %nice %system %iowait  %steal   %idle
>           22.81    0.00   45.28    3.59    0.00   28.32
>
> Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
> sda               2.80        25.60        80.00        128        400
> sdb            1073.60     45891.20    203473.60     229456    1017368
>
> avg-cpu:  %user   %nice %system %iowait  %steal   %idle
>           17.28    0.00   74.49    0.32    0.00    7.92
>
> Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
> sda               3.44         0.00        83.97          0        440
> sdb            1174.62     52370.99    209789.31     274424    1099296
>
> avg-cpu:  %user   %nice %system %iowait  %steal   %idle
>           51.72    0.00   47.60    0.31    0.00    0.38
>
> Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
> sda               1.20         0.00        22.40          0        112
> sdb            1094.20     67492.80    177187.20     337464     885936
>
> avg-cpu:  %user   %nice %system %iowait  %steal   %idle
>           43.73    0.00   36.19    3.03    0.00   17.05
>
> Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
> sda               1.80         0.00        46.40          0        232
> sdb            1241.20    100969.60    162806.40     504848     814032
>
> avg-cpu:  %user   %nice %system %iowait  %steal   %idle
>           37.09    0.00   58.61    0.77    0.00    3.54
>
> Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
> sda               4.60       182.40        19.20        912         96
> sdb            1235.20     47780.80    235912.00     238904    1179560
>
> avg-cpu:  %user   %nice %system %iowait  %steal   %idle
>           47.23    0.00   42.49    3.09    0.00    7.19
>
> Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
> sda               1.60         0.00        46.40          0        232
> sdb            1005.20     86502.40    135886.40     432512     679432
>
>
>
> ---------------------------
> Thanks & Regards
> Himanish
>

--
Thanks & Regards
Himanish