Hi,
Is it possible to get the execution time of the constituent map/reduce
tasks of a MapReduce job (say sort) at the end of a job run?
Preferably, can we obtain this programatically?

Thanks,
Bikash

On 04/04/2012 11:20 PM, bikash sharma wrote:
> Hi,
> Is it possible to get the execution time of the constituent map/reduce
> tasks of a MapReduce job (say sort) at the end of a job run?
> Preferably, can we obtain this programatically?
>
> Thanks,
> Bikash
>

You could parse the logs inside the job's output directory (<output
dir>/_logs)

You may also find this tool to be interesting:
http://www.cs.duke.edu/starfish/

It can extract all that task info for you, after doing a profiling run.

--
Luca Pireddu
CRS4 - Distributed Computing Group
Loc. Pixina Manna Edificio 1
09010 Pula (CA), Italy
Tel: +39 0709250452

Hi,

Am 05.04.2012 um 00:20 schrieb bikash sharma:

> Is it possible to get the execution time of the constituent map/reduce
> tasks of a MapReduce job (say sort) at the end of a job run?
> Preferably, can we obtain this programatically?
you can access the JobTracker's web UI and see the start and stop timestamps for every individual task.

Since the JobTracker Java API is exposed, you can write your own application to fetch that data through your own code.

Also, "hadoop job" on the command line can be used to read job statistics.

Kai
--
Kai Voigt
[EMAIL PROTECTED]

Thanks Kai, I will try those.

On Thu, Apr 5, 2012 at 3:15 AM, Kai Voigt <[EMAIL PROTECTED]> wrote:

> Hi,
>
> Am 05.04.2012 um 00:20 schrieb bikash sharma:
>
> > Is it possible to get the execution time of the constituent map/reduce
> > tasks of a MapReduce job (say sort) at the end of a job run?
> > Preferably, can we obtain this programatically?
>
>
> you can access the JobTracker's web UI and see the start and stop
> timestamps for every individual task.
>
> Since the JobTracker Java API is exposed, you can write your own
> application to fetch that data through your own code.
>
> Also, "hadoop job" on the command line can be used to read job statistics.
>
> Kai
>
>
> --
> Kai Voigt
> [EMAIL PROTECTED]
>
>
>
>
>

How can "hadoop job" be used to read m/r statistics ?

On Thu, Apr 5, 2012 at 7:30 AM, bikash sharma <[EMAIL PROTECTED]>wrote:

> Thanks Kai, I will try those.
>
> On Thu, Apr 5, 2012 at 3:15 AM, Kai Voigt <[EMAIL PROTECTED]> wrote:
>
> > Hi,
> >
> > Am 05.04.2012 um 00:20 schrieb bikash sharma:
> >
> > > Is it possible to get the execution time of the constituent map/reduce
> > > tasks of a MapReduce job (say sort) at the end of a job run?
> > > Preferably, can we obtain this programatically?
> >
> >
> > you can access the JobTracker's web UI and see the start and stop
> > timestamps for every individual task.
> >
> > Since the JobTracker Java API is exposed, you can write your own
> > application to fetch that data through your own code.
> >
> > Also, "hadoop job" on the command line can be used to read job
> statistics.
> >
> > Kai
> >
> >
> > --
> > Kai Voigt
> > [EMAIL PROTECTED]
> >
> >
> >
> >
> >
>

--
Jay Vyas
MMSB/UCHC

(excuse the typo in the last email : I meant "I've been playing with Cinch"
, not "I've been with Cinch")....

On Thu, Apr 5, 2012 at 7:54 AM, Jay Vyas <[EMAIL PROTECTED]> wrote:

> How can "hadoop job" be used to read m/r statistics ?
>
> On Thu, Apr 5, 2012 at 7:30 AM, bikash sharma <[EMAIL PROTECTED]>wrote:
>
>> Thanks Kai, I will try those.
>>
>> On Thu, Apr 5, 2012 at 3:15 AM, Kai Voigt <[EMAIL PROTECTED]> wrote:
>>
>> > Hi,
>> >
>> > Am 05.04.2012 um 00:20 schrieb bikash sharma:
>> >
>> > > Is it possible to get the execution time of the constituent map/reduce
>> > > tasks of a MapReduce job (say sort) at the end of a job run?
>> > > Preferably, can we obtain this programatically?
>> >
>> >
>> > you can access the JobTracker's web UI and see the start and stop
>> > timestamps for every individual task.
>> >
>> > Since the JobTracker Java API is exposed, you can write your own
>> > application to fetch that data through your own code.
>> >
>> > Also, "hadoop job" on the command line can be used to read job
>> statistics.
>> >
>> > Kai
>> >
>> >
>> > --
>> > Kai Voigt
>> > [EMAIL PROTECTED]
>> >
>> >
>> >
>> >
>> >
>>
>
>
>
> --
> Jay Vyas
> MMSB/UCHC
>

--
Jay Vyas
MMSB/UCHC

Yes, how can we use "hadoop job" to get MR job stats, especially
constituent task finish times?
On Thu, Apr 5, 2012 at 9:02 AM, Jay Vyas <[EMAIL PROTECTED]> wrote:

> (excuse the typo in the last email : I meant "I've been playing with Cinch"
> , not "I've been with Cinch")....
>
> On Thu, Apr 5, 2012 at 7:54 AM, Jay Vyas <[EMAIL PROTECTED]> wrote:
>
> > How can "hadoop job" be used to read m/r statistics ?
> >
> > On Thu, Apr 5, 2012 at 7:30 AM, bikash sharma <[EMAIL PROTECTED]
> >wrote:
> >
> >> Thanks Kai, I will try those.
> >>
> >> On Thu, Apr 5, 2012 at 3:15 AM, Kai Voigt <[EMAIL PROTECTED]> wrote:
> >>
> >> > Hi,
> >> >
> >> > Am 05.04.2012 um 00:20 schrieb bikash sharma:
> >> >
> >> > > Is it possible to get the execution time of the constituent
> map/reduce
> >> > > tasks of a MapReduce job (say sort) at the end of a job run?
> >> > > Preferably, can we obtain this programatically?
> >> >
> >> >
> >> > you can access the JobTracker's web UI and see the start and stop
> >> > timestamps for every individual task.
> >> >
> >> > Since the JobTracker Java API is exposed, you can write your own
> >> > application to fetch that data through your own code.
> >> >
> >> > Also, "hadoop job" on the command line can be used to read job
> >> statistics.
> >> >
> >> > Kai
> >> >
> >> >
> >> > --
> >> > Kai Voigt
> >> > [EMAIL PROTECTED]
> >> >
> >> >
> >> >
> >> >
> >> >
> >>
> >
> >
> >
> > --
> > Jay Vyas
> > MMSB/UCHC
> >
>
>
>
> --
> Jay Vyas
> MMSB/UCHC
>