Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase, mail # user - Force the number of map tasks in MR?


Copy link to this message
-
Re: Force the number of map tasks in MR?
Kevin O'dell 2012-10-11, 20:36
J-M,

  It should be in the mapred-site.xml the values
are mapred.tasktracker.map.tasks.maximum and
mapred.tasktracker.reduce.tasks.maximum.  This is the default in CDH4

<property>
  <name>mapreduce.tasktracker.map.tasks.maximum</name>
  <value>2</value>
  <description>The maximum number of map tasks that will be run
  simultaneously by a task tracker.
  </description>
</property>

<property>
  <name>mapreduce.tasktracker.reduce.tasks.maximum</name>
  <value>2</value>
  <description>The maximum number of reduce tasks that will be run
  simultaneously by a task tracker.
  </description>
</property>

This would explain why they are going 2 by 2.  Does this help?

On Thu, Oct 11, 2012 at 4:25 PM, Jean-Marc Spaggiari <
[EMAIL PROTECTED]> wrote:

> I don't know. I did not touched that. Where can I found this information?
>
> 2012/10/11 Kevin O'dell <[EMAIL PROTECTED]>:
> > What are you max tasks set to?
> >
> > On Thu, Oct 11, 2012 at 3:59 PM, Jean-Marc Spaggiari <
> > [EMAIL PROTECTED]> wrote:
> >
> >> Hi,
> >>
> >> Is there a way to force the number of map tasks in a MR?
> >>
> >> I have a 25 regions table splitted over 6 nodes. But the MR is running
> >> the tasks only 2 by 2.
> >>
> >> Is there a way to force it to run one task on each regionserver
> >> serving at least one region? Why is the MR waiting for 2 taskes to
> >> complete before sending to the other tasks?
> >>
> >> I'm starting the MR with a caching of 100.
> >>
> >> I tried mapred.map.tasks and speculative=false with no success.
> >>
> >> Any idea how I can increase it this number of tasks?
> >>
> >> Thanks,
> >>
> >> JM
> >>
> >
> >
> >
> > --
> > Kevin O'Dell
> > Customer Operations Engineer, Cloudera
>

--
Kevin O'Dell
Customer Operations Engineer, Cloudera