I have a question that I got an interesting and helpful answer for on the IRC channel today, but thought I'd open it up to a larger group as well.

   My problem is hopefully a very common problem.  I'm using Ganglia for trend graphing.  We are using Ganglia 3.0.7 for convenience since RPMS are available for CentOS 5.4 through EPEL.  Everything generally works great, but Ganglia doesn't appear to gather Disk I/O stats ( writes/sec, reads/sec, etc ) as a metric.  Some quick looking on the web seemed to confirm that ganglia doesn't capture disk I/O stats by default, but talk about how it's generally pretty easy to capture using gmetric.

   I guess my question would be is there a common way of doing this?  What do you use?  Do you find Disk I/O to be a useful/necessary metric?

   Any thoughts would be much appreciated.

I have found iotop to be useful -    http://guichaz.free.fr/iotop/  .
Andy Sautins wrote:
>     I have a question that I got an interesting and helpful answer for on the IRC channel today, but thought I'd open it up to a larger group as well.
>
>    My problem is hopefully a very common problem.  I'm using Ganglia for trend graphing.  We are using Ganglia 3.0.7 for convenience since RPMS are available for CentOS 5.4 through EPEL.  Everything generally works great, but Ganglia doesn't appear to gather Disk I/O stats ( writes/sec, reads/sec, etc ) as a metric.  Some quick looking on the web seemed to confirm that ganglia doesn't capture disk I/O stats by default, but talk about how it's generally pretty easy to capture using gmetric.
>
>    I guess my question would be is there a common way of doing this?  What do you use?  Do you find Disk I/O to be a useful/necessary metric?
>
>    Any thoughts would be much appreciated.
>
>
>

On 12/10/2009 04:55 PM, Andy Sautins wrote:
>
>      I have a question that I got an interesting and helpful answer for on the IRC channel today, but thought I'd open it up to a larger group as well.
>
>     My problem is hopefully a very common problem.  I'm using Ganglia for trend graphing.  We are using Ganglia 3.0.7 for convenience since RPMS are available for CentOS 5.4 through EPEL.  Everything generally works great, but Ganglia doesn't appear to gather Disk I/O stats ( writes/sec, reads/sec, etc ) as a metric.  Some quick looking on the web seemed to confirm that ganglia doesn't capture disk I/O stats by default, but talk about how it's generally pretty easy to capture using gmetric.
>
>     I guess my question would be is there a common way of doing this?  What do you use?  Do you find Disk I/O to be a useful/necessary metric?
>
>     Any thoughts would be much appreciated.

In the past we've used a wrapper around 'iostat -k /dev/XXX 60' to
report disk io statistics to our internal reporting tools.

--Mike

If you're looking for ganglia gmetric scripts for Disk I/O, take a look at
http://ganglia.info/gmetric/ or http://ben.hartshorne.net/ganglia/.  At the
very bottom of Ben's page you'll find disk_metric.sh.

-Matt

On Thu, Dec 10, 2009 at 4:55 PM, Andy Sautins
<[EMAIL PROTECTED]>wrote:

>
>    I have a question that I got an interesting and helpful answer for on
> the IRC channel today, but thought I'd open it up to a larger group as well.
>
>   My problem is hopefully a very common problem.  I'm using Ganglia for
> trend graphing.  We are using Ganglia 3.0.7 for convenience since RPMS are
> available for CentOS 5.4 through EPEL.  Everything generally works great,
> but Ganglia doesn't appear to gather Disk I/O stats ( writes/sec, reads/sec,
> etc ) as a metric.  Some quick looking on the web seemed to confirm that
> ganglia doesn't capture disk I/O stats by default, but talk about how it's
> generally pretty easy to capture using gmetric.
>
>   I guess my question would be is there a common way of doing this?  What
> do you use?  Do you find Disk I/O to be a useful/necessary metric?
>
>   Any thoughts would be much appreciated.
>
>

If you want rich detailed metrics from the OS, and hardware such as IO rates, I would HIGHLY recommend SGI's open-source Performance Co-Pilot (PCP)

http://oss.sgi.com/projects/pcp/

It really hasn't been given the visibility many other projects have but having used this for 5 years now I can't live with out it.  SGI are obviously large scale, and PCP was designed from the ground up to deal with high volumes, it works for NASA's Columbia:

http://www.nas.nasa.gov/About/Projects/Columbia/columbia.html

You can capture archives of data and go over it retrospectively in fine detail at any point, looking for correlation between metrics.

cheers,

Paul
On 11/12/2009, at 12:40 PM, Matt Massie wrote:

> If you're looking for ganglia gmetric scripts for Disk I/O, take a look at
> http://ganglia.info/gmetric/ or http://ben.hartshorne.net/ganglia/.  At the
> very bottom of Ben's page you'll find disk_metric.sh.
>
> -Matt
>
> On Thu, Dec 10, 2009 at 4:55 PM, Andy Sautins
> <[EMAIL PROTECTED]>wrote:
>
>>
>>   I have a question that I got an interesting and helpful answer for on
>> the IRC channel today, but thought I'd open it up to a larger group as well.
>>
>>  My problem is hopefully a very common problem.  I'm using Ganglia for
>> trend graphing.  We are using Ganglia 3.0.7 for convenience since RPMS are
>> available for CentOS 5.4 through EPEL.  Everything generally works great,
>> but Ganglia doesn't appear to gather Disk I/O stats ( writes/sec, reads/sec,
>> etc ) as a metric.  Some quick looking on the web seemed to confirm that
>> ganglia doesn't capture disk I/O stats by default, but talk about how it's
>> generally pretty easy to capture using gmetric.
>>
>>  I guess my question would be is there a common way of doing this?  What
>> do you use?  Do you find Disk I/O to be a useful/necessary metric?
>>
>>  Any thoughts would be much appreciated.
>>
>>

On Mon, Dec 14, 2009 at 5:59 PM, Paul Smith <[EMAIL PROTECTED]> wrote:
> If you want rich detailed metrics from the OS, and hardware such as IO rates, I would HIGHLY recommend SGI's open-source Performance Co-Pilot (PCP)
>
> http://oss.sgi.com/projects/pcp/
>
> It really hasn't been given the visibility many other projects have but having used this for 5 years now I can't live with out it.  SGI are obviously large scale, and PCP was designed from the ground up to deal with high volumes, it works for NASA's Columbia:
>
> http://www.nas.nasa.gov/About/Projects/Columbia/columbia.html
>
> You can capture archives of data and go over it retrospectively in fine detail at any point, looking for correlation between metrics.
>
> cheers,
>
> Paul
> On 11/12/2009, at 12:40 PM, Matt Massie wrote:
>
>> If you're looking for ganglia gmetric scripts for Disk I/O, take a look at
>> http://ganglia.info/gmetric/ or http://ben.hartshorne.net/ganglia/.  At the
>> very bottom of Ben's page you'll find disk_metric.sh.
>>
>> -Matt
>>
>> On Thu, Dec 10, 2009 at 4:55 PM, Andy Sautins
>> <[EMAIL PROTECTED]>wrote:
>>
>>>
>>>   I have a question that I got an interesting and helpful answer for on
>>> the IRC channel today, but thought I'd open it up to a larger group as well.
>>>
>>>  My problem is hopefully a very common problem.  I'm using Ganglia for
>>> trend graphing.  We are using Ganglia 3.0.7 for convenience since RPMS are
>>> available for CentOS 5.4 through EPEL.  Everything generally works great,
>>> but Ganglia doesn't appear to gather Disk I/O stats ( writes/sec, reads/sec,
>>> etc ) as a metric.  Some quick looking on the web seemed to confirm that
>>> ganglia doesn't capture disk I/O stats by default, but talk about how it's
>>> generally pretty easy to capture using gmetric.
>>>
>>>  I guess my question would be is there a common way of doing this?  What
>>> do you use?  Do you find Disk I/O to be a useful/necessary metric?
>>>
>>>  Any thoughts would be much appreciated.
>>>
>>>
>
>

A modern (linux/unix) operating system provides this information for
each disk via SNMP:

I plot both the bytes read/written and # of transactions for each
disk. Be warned if you are doing very very very heavy disk usage this
value can wrap around in a 5 minute polling interval, you might have
to go to a 1 minute interval.

 <oid_index>.1.3.6.1.4.1.2021.13.15.1.1.1</oid_index>

        <fields>
                <hrDiskIOIndex>
                        <name>Index</name>
                        <method>walk</method>
                        <source>value</source>
                        <direction>input</direction>
                        <oid>.1.3.6.1.4.1.2021.13.15.1.1.1</oid>
                </hrDiskIOIndex>
                <hrDiskIODescr>
                        <name>Description</name>
                        <method>walk</method>
                        <source>value</source>
                        <direction>input</direction>
                        <oid>.1.3.6.1.4.1.2021.13.15.1.1.2</oid>
                </hrDiskIODescr>
                <hrDiskIONRead>
                        <name>Number of Bytes Read</name>
                        <method>walk</method>
                        <source>value</source>
                        <direction>output</direction>
                        <oid>.1.3.6.1.4.1.2021.13.15.1.1.3</oid>
                </hrDiskIONRead>
                <hrDiskIONWrite>
                        <name>DiskIO Number of Bytes Written</name>
                        <method>walk</method>
                        <source>value</source>
                        <direction>output</direction>
                        <oid>.1.3.6.1.4.1.2021.13.15.1.1.4</oid>
                </hrDiskIONWrite>
                <hrDiskIOReads>
                        <name>Number of Read accesses</name>
                        <method>walk</method>
                        <source>value</source>
                        <direction>output</direction>
                        <oid>.1.3.6.1.4.1.2021.13.15.1.1.5</oid>
                </hrDiskIOReads>
                <hrDiskIOWrites>
                        <name>Number of Write accesses</name>
                        <method>walk</method>
                        <source>value</source>
                        <direction>output</direction>
                        <oid>.1.3.6.1.4.1.2021.13.15.1.1.6</oid>
                </hrDiskIOWrites>
        </fields>

Edward