Hi all,

I'd like to log and monitor average job completion times on hadoop.
(For example, wall time from the perspective of the JobTracker)
What might be the best way to do this?

I see a lot of metrics exported through the metrics2 interface:
are job completion times part of this? Is there documentation I could read
to understand the exported jobtracker/tasktracker metrics?

Thanks a lot!
--
Bharath Ravi

Hi Bharath
        You could be getting detailed metrics on a per job basis from the jobtracker web UI

http://<NameNodeHost>:50030/jobtracker.jsp

If you drill down you can get still further details on task levels .

Regards
Bejoy K S

>From handheld, Please excuse typos.

-----Original Message-----
From: Bharath Ravi <[EMAIL PROTECTED]>
Date: Sun, 4 Mar 2012 17:39:01
To: <[EMAIL PROTECTED]>
Reply-To: [EMAIL PROTECTED]
Subject: Tracking Job completion times

Hi all,

I'd like to log and monitor average job completion times on hadoop.
(For example, wall time from the perspective of the JobTracker)
What might be the best way to do this?

I see a lot of metrics exported through the metrics2 interface:
are job completion times part of this? Is there documentation I could read
to understand the exported jobtracker/tasktracker metrics?

Thanks a lot!
--
Bharath Ravi

Thanks a lot Bejoy!
The Web UI does give me start and finish times, but I was wondering if
there is
a way to access these stats through an API, without having to grep through
HTML.

The "hadoop jobs -status" API was useful, but it doesn't seem to list wall
completion times.
(It does give me CPU time though). Am I missing something?

On 4 March 2012 17:46, <[EMAIL PROTECTED]> wrote:

> **
> Hi Bharath
> You could be getting detailed metrics on a per job basis from the
> jobtracker web UI
>
> http://<NameNodeHost>:50030/jobtracker.jsp
>
> If you drill down you can get still further details on task levels .
> Regards
> Bejoy K S
>
> From handheld, Please excuse typos.
> ------------------------------
> *From: * Bharath Ravi <[EMAIL PROTECTED]>
> *Date: *Sun, 4 Mar 2012 17:39:01 -0500
> *To: *<[EMAIL PROTECTED]>
> *ReplyTo: * [EMAIL PROTECTED]
> *Subject: *Tracking Job completion times
>
> Hi all,
>
> I'd like to log and monitor average job completion times on hadoop.
> (For example, wall time from the perspective of the JobTracker)
> What might be the best way to do this?
>
> I see a lot of metrics exported through the metrics2 interface:
> are job completion times part of this? Is there documentation I could read
> to understand the exported jobtracker/tasktracker metrics?
>
> Thanks a lot!
> --
> Bharath Ravi
>
>
--
Bharath Ravi

Bharath,

Try the hadoop job -history API

On 2012/03/05 8:06, Bharath Ravi wrote:
> The Web UI does give me start and finish times, but I was wondering if
> there is
> a way to access these stats through an API, without having to grep
> through HTML.
>
> The "hadoop jobs -status" API was useful, but it doesn't seem to list
> wall completion times.
> (It does give me CPU time though). Am I missing something?

In terms of accessing metrics(2) programmatically, do people generally extend FileSink for collecting data from small (1 - 15 node) installations? As opposed to using Chukwa, etc.
C
On Mar 4, 2012, at 7:08 PM, George Datskos wrote:

> Bharath,
>
> Try the hadoop job -history API
>
>
>
> On 2012/03/05 8:06, Bharath Ravi wrote:
>> The Web UI does give me start and finish times, but I was wondering if there is
>> a way to access these stats through an API, without having to grep through HTML.
>>
>> The "hadoop jobs -status" API was useful, but it doesn't seem to list wall completion times.
>> (It does give me CPU time though). Am I missing something?
>
>

Thanks all!
Trying out the history API, I ran into a spot of trouble: I'd like to log
history to a custom location
(It is by default the job output dir in DFS).

However, setting hadoop.job.history.location (I tried adding it to the
core- and mapred-site xml's)
doesn't seem to help. The path it points to exists on disk.
What could I be missing?

On 5 March 2012 16:27, Charles Earl <[EMAIL PROTECTED]> wrote:

> In terms of accessing metrics(2) programmatically, do people generally
> extend FileSink for collecting data from small (1 - 15 node) installations?
> As opposed to using Chukwa, etc.
> C
> On Mar 4, 2012, at 7:08 PM, George Datskos wrote:
>
> > Bharath,
> >
> > Try the hadoop job -history API
> >
> >
> >
> > On 2012/03/05 8:06, Bharath Ravi wrote:
> >> The Web UI does give me start and finish times, but I was wondering if
> there is
> >> a way to access these stats through an API, without having to grep
> through HTML.
> >>
> >> The "hadoop jobs -status" API was useful, but it doesn't seem to list
> wall completion times.
> >> (It does give me CPU time though). Am I missing something?
> >
> >
>
>
--
Bharath Ravi