Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
MapReduce >> mail # user >> I/O stats interpretation during concurrent hive M/R runs


+
Himanish Kushary 2012-08-23, 19:14
Copy link to this message
-
Re: I/O stats interpretation during concurrent hive M/R runs
After sending this message I issued the iostat -dxm 5 command on the
DNs.... the %util column shows 70-80  average value sometimes going up to
90-100 for few seconds

Does this mean the disk is becoming the bottleneck ? or is this normal ?

On Thu, Aug 23, 2012 at 3:14 PM, Himanish Kushary <[EMAIL PROTECTED]>wrote:

> Hi,
>
> I am curious about interpretation of the output from iostat on a datanode
> during a M/R run.I want to understand how to diagnosis a disk i/o issue in
> a hadoop cluster.
>
> Is there any good documentation to help me understand the results from
> iostats in Hadoop context ?
>
> Here are the iostat output  from a DN while two intensive M/R jobs were
> executing.Does this result indicate any performance issue related to the
> disks ?
>
> Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
> sda               1.80         0.00        59.20          0        296
> sdb            1436.20     96376.00    211424.00     481880    1057120
>
> avg-cpu:  %user   %nice %system %iowait  %steal   %idle
>           22.81    0.00   45.28    3.59    0.00   28.32
>
> Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
> sda               2.80        25.60        80.00        128        400
> sdb            1073.60     45891.20    203473.60     229456    1017368
>
> avg-cpu:  %user   %nice %system %iowait  %steal   %idle
>           17.28    0.00   74.49    0.32    0.00    7.92
>
> Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
> sda               3.44         0.00        83.97          0        440
> sdb            1174.62     52370.99    209789.31     274424    1099296
>
> avg-cpu:  %user   %nice %system %iowait  %steal   %idle
>           51.72    0.00   47.60    0.31    0.00    0.38
>
> Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
> sda               1.20         0.00        22.40          0        112
> sdb            1094.20     67492.80    177187.20     337464     885936
>
> avg-cpu:  %user   %nice %system %iowait  %steal   %idle
>           43.73    0.00   36.19    3.03    0.00   17.05
>
> Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
> sda               1.80         0.00        46.40          0        232
> sdb            1241.20    100969.60    162806.40     504848     814032
>
> avg-cpu:  %user   %nice %system %iowait  %steal   %idle
>           37.09    0.00   58.61    0.77    0.00    3.54
>
> Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
> sda               4.60       182.40        19.20        912         96
> sdb            1235.20     47780.80    235912.00     238904    1179560
>
> avg-cpu:  %user   %nice %system %iowait  %steal   %idle
>           47.23    0.00   42.49    3.09    0.00    7.19
>
> Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
> sda               1.60         0.00        46.40          0        232
> sdb            1005.20     86502.40    135886.40     432512     679432
>
>
>
> ---------------------------
> Thanks & Regards
> Himanish
>

--
Thanks & Regards
Himanish
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB