Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce >> mail # user >> Profiling Hadoop Map Reduce with the 20.2 API


Copy link to this message
-
Re: Profiling Hadoop Map Reduce with the 20.2 API
Hi,

>   I recently started developing with Hadoop using the 20.2 API.  I'm looking
> to profile one of my jobs but I haven't been able to find any documentation
> about how to do this.  For the earlier (deprecated) API, there's some
> documentation on how to profile with the JobConf class
> (http://hadoop.apache.org/common/docs/current/mapred_tutorial.html#Profiling).
> Is there anything equivalent in 20.2, or is there another process to use for
> it?  Search engines and javadoc searching haven't turned up anything so far
> for me.

All APIs in the deprecated JobConf class are wrappers around the
Configuration API that you probably are using in the 20.2 new API. For
e.g. :

setProfileEnabled = configuration.setBoolean("mapred.task.profile", value)
setProfileTaskRange = configuration.set("mapred.task.profile.maps",
value) or configuration.set("mapred.task.profile.reduces", value)
setProfileParams = configuration.set("mapred.task.profile.params", value)

Could you try setting these parameters in the Configuration you might
be using for the Job and see if that works ?

Thanks
Hemanth
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB