Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
HBase >> mail # user >> Force the number of map tasks in MR?


+
Jean-Marc Spaggiari 2012-10-11, 19:59
+
Kevin Odell 2012-10-11, 20:23
+
Jean-Marc Spaggiari 2012-10-11, 20:25
+
Kevin Odell 2012-10-11, 20:36
Copy link to this message
-
Re: Force the number of map tasks in MR?
But this is the limit per tasktracker, right?

And I have 6 nodes, so 6 tasktrackers, which mean it should go up to 12 tasks?

Take a look at 2.7 here: http://wiki.apache.org/hadoop/FAQ

I just tried with the setting below (changing 2 by 6) but I'm getting
the same result.

JM

2012/10/11 Kevin O'dell <[EMAIL PROTECTED]>:
> J-M,
>
>   It should be in the mapred-site.xml the values
> are mapred.tasktracker.map.tasks.maximum and
> mapred.tasktracker.reduce.tasks.maximum.  This is the default in CDH4
>
> <property>
>   <name>mapreduce.tasktracker.map.tasks.maximum</name>
>   <value>2</value>
>   <description>The maximum number of map tasks that will be run
>   simultaneously by a task tracker.
>   </description>
> </property>
>
> <property>
>   <name>mapreduce.tasktracker.reduce.tasks.maximum</name>
>   <value>2</value>
>   <description>The maximum number of reduce tasks that will be run
>   simultaneously by a task tracker.
>   </description>
> </property>
>
> This would explain why they are going 2 by 2.  Does this help?
>
> On Thu, Oct 11, 2012 at 4:25 PM, Jean-Marc Spaggiari <
> [EMAIL PROTECTED]> wrote:
>
>> I don't know. I did not touched that. Where can I found this information?
>>
>> 2012/10/11 Kevin O'dell <[EMAIL PROTECTED]>:
>> > What are you max tasks set to?
>> >
>> > On Thu, Oct 11, 2012 at 3:59 PM, Jean-Marc Spaggiari <
>> > [EMAIL PROTECTED]> wrote:
>> >
>> >> Hi,
>> >>
>> >> Is there a way to force the number of map tasks in a MR?
>> >>
>> >> I have a 25 regions table splitted over 6 nodes. But the MR is running
>> >> the tasks only 2 by 2.
>> >>
>> >> Is there a way to force it to run one task on each regionserver
>> >> serving at least one region? Why is the MR waiting for 2 taskes to
>> >> complete before sending to the other tasks?
>> >>
>> >> I'm starting the MR with a caching of 100.
>> >>
>> >> I tried mapred.map.tasks and speculative=false with no success.
>> >>
>> >> Any idea how I can increase it this number of tasks?
>> >>
>> >> Thanks,
>> >>
>> >> JM
>> >>
>> >
>> >
>> >
>> > --
>> > Kevin O'Dell
>> > Customer Operations Engineer, Cloudera
>>
>
>
>
> --
> Kevin O'Dell
> Customer Operations Engineer, Cloudera
+
Jonathan Bishop 2012-10-11, 20:50
+
Bryan Beaudreault 2012-10-12, 01:10
+
Jean-Marc Spaggiari 2012-10-12, 01:45
+
Kevin Odell 2012-10-11, 20:48