Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # user >> Obtaining the number of map slots through the API (Hadoop 0.20.2)


Copy link to this message
-
Re: Obtaining the number of map slots through the API (Hadoop 0.20.2)
Hi,

>
> The optimization of one Hadoop job I'm running would benefit from knowing
> the
> maximum number of map slots in the Hadoop cluster.
>
> This number can be obtained (if my understanding is correct) by:
>
> * parsing the mapred-site.xml file to get
>  the mapred.tasktracker.map.tasks.maximum value (assuming it is set of
> course)
>
> * parsing the slaves file to get the maximum number of compute nodes in the
> cluster
>
> * multiplying the 2 values
>
> My question is:
> I would like to learn about *all* possible ways to get this information
> through API calls (either the Hadoop Common API or the Hadoop MapReduce
> API), i.e. obtaining it through a Job object, through a Configuration
> object,...
>

The easiest way I can think of is using
o.a.h.m.ClusterStatus.getMaxMapTasks(). You can get an instance to
ClusterStatus using JobClient.getClusterStatus().

Thanks
hemanth
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB