Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
MapReduce, mail # user - Profiling Hadoop Map Reduce with the 20.2 API

David Jurgens 2010-08-12, 22:33
Copy link to this message
Re: Profiling Hadoop Map Reduce with the 20.2 API
Hemanth Yamijala 2010-08-13, 05:16

>   I recently started developing with Hadoop using the 20.2 API.  I'm looking
> to profile one of my jobs but I haven't been able to find any documentation
> about how to do this.  For the earlier (deprecated) API, there's some
> documentation on how to profile with the JobConf class
> (http://hadoop.apache.org/common/docs/current/mapred_tutorial.html#Profiling).
> Is there anything equivalent in 20.2, or is there another process to use for
> it?  Search engines and javadoc searching haven't turned up anything so far
> for me.

All APIs in the deprecated JobConf class are wrappers around the
Configuration API that you probably are using in the 20.2 new API. For
e.g. :

setProfileEnabled = configuration.setBoolean("mapred.task.profile", value)
setProfileTaskRange = configuration.set("mapred.task.profile.maps",
value) or configuration.set("mapred.task.profile.reduces", value)
setProfileParams = configuration.set("mapred.task.profile.params", value)

Could you try setting these parameters in the Configuration you might
be using for the Job and see if that works ?

David Jurgens 2010-08-16, 20:34
Hemanth Yamijala 2010-08-17, 06:19