-Re: Help: How to increase amont maptasks per job ?
Ted Yu 2011-01-07, 21:19
Set higher values for mapred.tasktracker.map.tasks.maximum (and
mapred.tasktracker.reduce.tasks.maximum) in mapred-site.xml
On Fri, Jan 7, 2011 at 12:58 PM, Tali K <[EMAIL PROTECTED]> wrote:
> We have a jobs which runs in several map/reduce stages. In the first job,
> a large number of map tasks -82 are initiated, as expected.
> And that cause all nodes to be used.
> In a
> later job, where we are still dealing with large amounts of
> data, only 4 map tasks are initiated, and that caused to use only 4 nodes.
> This stage is actually the
> workhorse of the job, and requires much more processing power than the
> initial stage.
> We are trying to understand why only a few map tasks are
> being used, as we are not getting the full advantage of our cluster.