Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce, mail # user - Setting #Reducers at runtime


Copy link to this message
-
Re: Setting #Reducers at runtime
E. Sammer 2010-02-18, 16:37
On 2/18/10 11:24 AM, Rob Stewart wrote:
> Hi Eric, thanks.
>
> It appears not:
> ----------------
>   JobConf jobConf = new JobConf(getConf(), Sort.class);
>      jobConf.setJobName("join");
>
>      jobConf.setMapperClass(IdentityMapper.class);
>      jobConf.setReducerClass(IdentityReducer.class);
>
>      JobClient client = new JobClient(jobConf);
>      ClusterStatus cluster = client.getClusterStatus();
>      int num_maps = cluster.getTaskTrackers() *
>                     jobConf.getInt("test.sort.maps_per_host", 10);
>      int num_reduces = (int) (cluster.getMaxReduceTasks() * 0.9);
>      String sort_reduces = jobConf.get("test.sort.reduces_per_host");
>      if (sort_reduces != null) {
>         num_reduces = cluster.getTaskTrackers() *
>                         Integer.parseInt(sort_reduces);
>      }
>
> jobConf.setNumReduceTasks(num_reduces);
>
> -----------
>
> Any idea why my parameter for reduce tasks is being ignored ?

Rob:

It is setting the number of reducers itself. See the line:

jobConf.setNumReduceTasks(num_reduces)

In short, you can't control the number of reducers this code uses from
the command line.

--
Eric Sammer
[EMAIL PROTECTED]
http://esammer.blogspot.com