Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce >> mail # user >> Setting #Reducers at runtime


Copy link to this message
-
Re: Setting #Reducers at runtime
On 2/18/10 11:24 AM, Rob Stewart wrote:
> Hi Eric, thanks.
>
> It appears not:
> ----------------
>   JobConf jobConf = new JobConf(getConf(), Sort.class);
>      jobConf.setJobName("join");
>
>      jobConf.setMapperClass(IdentityMapper.class);
>      jobConf.setReducerClass(IdentityReducer.class);
>
>      JobClient client = new JobClient(jobConf);
>      ClusterStatus cluster = client.getClusterStatus();
>      int num_maps = cluster.getTaskTrackers() *
>                     jobConf.getInt("test.sort.maps_per_host", 10);
>      int num_reduces = (int) (cluster.getMaxReduceTasks() * 0.9);
>      String sort_reduces = jobConf.get("test.sort.reduces_per_host");
>      if (sort_reduces != null) {
>         num_reduces = cluster.getTaskTrackers() *
>                         Integer.parseInt(sort_reduces);
>      }
>
> jobConf.setNumReduceTasks(num_reduces);
>
> -----------
>
> Any idea why my parameter for reduce tasks is being ignored ?

Rob:

It is setting the number of reducers itself. See the line:

jobConf.setNumReduceTasks(num_reduces)

In short, you can't control the number of reducers this code uses from
the command line.

--
Eric Sammer
[EMAIL PROTECTED]
http://esammer.blogspot.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB