I am trying to gather the info regarding the amount of HDFS read/write for
each task in a given map-reduce job. How can I do that?

Let me add, I'd like to do this periodically to gather some performance
profile information.

On Tue, Apr 24, 2012 at 5:47 PM, Qu Chen <[EMAIL PROTECTED]> wrote:

> I am trying to gather the info regarding the amount of HDFS read/write for
> each task in a given map-reduce job. How can I do that?
>

Qu,

Every job has a history file that is, by default, stored under
$HADOOP_LOG_DIR/history.  These "job history" files list the amount of
hdfs read/write (and lots of other things) for every task.

On 2012/04/25 7:25, Qu Chen wrote:
> Let me add, I'd like to do this periodically to gather some
> performance profile information.
>
> On Tue, Apr 24, 2012 at 5:47 PM, Qu Chen <[EMAIL PROTECTED]
> <mailto:[EMAIL PROTECTED]>> wrote:
>
>     I am trying to gather the info regarding the amount of HDFS
>     read/write for each task in a given map-reduce job. How can I do that?
>
>

Hi Qu,

     You can access the HDFS read/write bytes for each task or job level using the below counters.

FileSystemCounters :  HDFS_BYTES_READ  
                                  FILE_BYTES_WRITTEN  

These can be accessed by using UI or API.

Thanks
Devaraj

________________________________________
From: George Datskos [[EMAIL PROTECTED]]
Sent: Wednesday, April 25, 2012 6:36 AM
To: [EMAIL PROTECTED]
Subject: Re: How to get the HDFS I/O information

Qu,

Every job has a history file that is, by default, stored under
$HADOOP_LOG_DIR/history.  These "job history" files list the amount of
hdfs read/write (and lots of other things) for every task.

On 2012/04/25 7:25, Qu Chen wrote:
> Let me add, I'd like to do this periodically to gather some
> performance profile information.
>
> On Tue, Apr 24, 2012 at 5:47 PM, Qu Chen <[EMAIL PROTECTED]
> <mailto:[EMAIL PROTECTED]>> wrote:
>
>     I am trying to gather the info regarding the amount of HDFS
>     read/write for each task in a given map-reduce job. How can I do that?
>
>

I am not sure if you want to use APIs, but to access through APIs, you can
use the following API in Counters.java:

 /**
   * Find a counter given the group and the name.
   * @param group the name of the group
   * @param name the internal name of the counter
   * @return the counter for that name
   */
  public synchronized Counter findCounter(String group, String name) {
...
}

For an example on how to use it, look at FileSystemStatisticUpdater in
Task.java. There are two more APIs, you can find in Counters.java.

Thanks,
Raj

On Wed, Apr 25, 2012 at 12:01 PM, Devaraj k <[EMAIL PROTECTED]> wrote:

> Hi Qu,
>
>     You can access the HDFS read/write bytes for each task or job level
> using the below counters.
>
> FileSystemCounters :  HDFS_BYTES_READ
>                                  FILE_BYTES_WRITTEN
>
> These can be accessed by using UI or API.
>
>
>
> Thanks
> Devaraj
>
> ________________________________________
> From: George Datskos [[EMAIL PROTECTED]]
> Sent: Wednesday, April 25, 2012 6:36 AM
> To: [EMAIL PROTECTED]
> Subject: Re: How to get the HDFS I/O information
>
> Qu,
>
> Every job has a history file that is, by default, stored under
> $HADOOP_LOG_DIR/history.  These "job history" files list the amount of
> hdfs read/write (and lots of other things) for every task.
>
> On 2012/04/25 7:25, Qu Chen wrote:
> > Let me add, I'd like to do this periodically to gather some
> > performance profile information.
> >
> > On Tue, Apr 24, 2012 at 5:47 PM, Qu Chen <[EMAIL PROTECTED]
> > <mailto:[EMAIL PROTECTED]>> wrote:
> >
> >     I am trying to gather the info regarding the amount of HDFS
> >     read/write for each task in a given map-reduce job. How can I do
> that?
> >
> >
>