Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Hadoop >> mail # user >> map task execution time


+
bikash sharma 2012-04-04, 21:20
Copy link to this message
-
Re: map task execution time
On 04/04/2012 11:20 PM, bikash sharma wrote:
> Hi,
> Is it possible to get the execution time of the constituent map/reduce
> tasks of a MapReduce job (say sort) at the end of a job run?
> Preferably, can we obtain this programatically?
>
> Thanks,
> Bikash
>

You could parse the logs inside the job's output directory (<output
dir>/_logs)

You may also find this tool to be interesting:
http://www.cs.duke.edu/starfish/

It can extract all that task info for you, after doing a profiling run.

--
Luca Pireddu
CRS4 - Distributed Computing Group
Loc. Pixina Manna Edificio 1
09010 Pula (CA), Italy
Tel: +39 0709250452
+
Kai Voigt 2012-04-05, 07:15
+
bikash sharma 2012-04-05, 12:30
+
Jay Vyas 2012-04-05, 12:54
+
Jay Vyas 2012-04-05, 13:02
+
bikash sharma 2012-04-05, 13:44