Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce >> mail # user >> Setting #Reducers at runtime


Copy link to this message
-
Re: Setting #Reducers at runtime
On 2/18/10 10:39 AM, Rob Stewart wrote:
> Hi there,
>
> I am using Hadoop 0.20.1 and I am trying to submit jobs to the cluster
> as jars.
>
> Here's what I'm trying to do:
>
>  > hadoop jar $hadoopPath/hadoop-*-examples.jar join
> -Dmapred.reduce.tasks=10 -inFormat
> org.apache.hadoop.mapred.KeyValueTextInputFormat  -outKey
> org.apache.hadoop.io.Text file1.dat file2.dat output.dat
>
>
> However, my parameter setting of attempting to state 10 reducers for the
> job is not being honoured by Hadoop, instead choosing some other number.
> Where am I going wrong here? I do not want to have to change this value
> in hadoop/conf.*.xml files, as I am attempting to show the expressive
> power of Hadoop. Note: The power to specify the number or reducers in
> possible in both Pig and Hive.
>
>
> Thanks,
>
>
> Rob Stewart

Rob:

It's possible that something inside the jar is calling
JobConf.setNumReducers(x) after it parses the command line args. That
would cause this type of behavior. I haven't looked at the source for
the join example to confirm this, though.

Regards.
--
Eric Sammer
[EMAIL PROTECTED]
http://esammer.blogspot.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB