Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Hadoop >> mail # user >> map task execution time


+
bikash sharma 2012-04-04, 21:20
Copy link to this message
-
Re: map task execution time
On 04/04/2012 11:20 PM, bikash sharma wrote:
> Hi,
> Is it possible to get the execution time of the constituent map/reduce
> tasks of a MapReduce job (say sort) at the end of a job run?
> Preferably, can we obtain this programatically?
>
> Thanks,
> Bikash
>

You could parse the logs inside the job's output directory (<output
dir>/_logs)

You may also find this tool to be interesting:
http://www.cs.duke.edu/starfish/

It can extract all that task info for you, after doing a profiling run.

--
Luca Pireddu
CRS4 - Distributed Computing Group
Loc. Pixina Manna Edificio 1
09010 Pula (CA), Italy
Tel: +39 0709250452
+
Kai Voigt 2012-04-05, 07:15
+
bikash sharma 2012-04-05, 12:30
+
Jay Vyas 2012-04-05, 12:54
+
Jay Vyas 2012-04-05, 13:02
+
bikash sharma 2012-04-05, 13:44
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB