Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
MapReduce >> mail # user >> I/O stats interpretation during concurrent hive M/R runs


Copy link to this message
-
I/O stats interpretation during concurrent hive M/R runs
Hi,

I am curious about interpretation of the output from iostat on a datanode
during a M/R run.I want to understand how to diagnosis a disk i/o issue in
a hadoop cluster.

Is there any good documentation to help me understand the results from
iostats in Hadoop context ?

Here are the iostat output  from a DN while two intensive M/R jobs were
executing.Does this result indicate any performance issue related to the
disks ?

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda               1.80         0.00        59.20          0        296
sdb            1436.20     96376.00    211424.00     481880    1057120

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          22.81    0.00   45.28    3.59    0.00   28.32

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda               2.80        25.60        80.00        128        400
sdb            1073.60     45891.20    203473.60     229456    1017368

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          17.28    0.00   74.49    0.32    0.00    7.92

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda               3.44         0.00        83.97          0        440
sdb            1174.62     52370.99    209789.31     274424    1099296

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          51.72    0.00   47.60    0.31    0.00    0.38

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda               1.20         0.00        22.40          0        112
sdb            1094.20     67492.80    177187.20     337464     885936

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          43.73    0.00   36.19    3.03    0.00   17.05

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda               1.80         0.00        46.40          0        232
sdb            1241.20    100969.60    162806.40     504848     814032

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          37.09    0.00   58.61    0.77    0.00    3.54

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda               4.60       182.40        19.20        912         96
sdb            1235.20     47780.80    235912.00     238904    1179560

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          47.23    0.00   42.49    3.09    0.00    7.19

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda               1.60         0.00        46.40          0        232
sdb            1005.20     86502.40    135886.40     432512     679432

---------------------------
Thanks & Regards
Himanish
+
Himanish Kushary 2012-08-23, 19:44