Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HDFS >> mail # user >> Increase number of map slots


Copy link to this message
-
Re: Increase number of map slots
> I think mapred.tasktracker.map.tasks.maximum sets the number of map
tasks and not slots.

This is incorrect. The property does configure slots. Please also see
http://wiki.apache.org/hadoop/HowManyMapsAndReduces and
http://wiki.apache.org/hadoop/FAQ#I_see_a_maximum_of_2_maps.2BAC8-reduces_spawned_concurrently_on_each_TaskTracker.2C_how_do_I_increase_that.3F
for more.

On Wed, Jun 6, 2012 at 9:56 AM, kartheek muthyala
<[EMAIL PROTECTED]> wrote:
> I think mapred.tasktracker.map.tasks.maximum sets the number of map
> tasks and not slots.
> Keith, so that means per core, if I specify the maximum number of map
> tasks it can host, whether all the maps per slot will run parallely or
> they execute one after the other?. I understand that maps that are
> scheduled on different cores can run parallely.
>
> Thanks,
> Kartheek.
>
> On Tue, Jun 5, 2012 at 11:05 AM, madhu phatak <[EMAIL PROTECTED]> wrote:
>> Hi,
>>  You can change the no of slots using the following properties in
>> mapred-site.xml.
>>
>>  mapred.tasktracker.map.tasks.maximum ( for map slots)
>>  mapred.tasktracker.reduce.tasks.maximum (for reduce slot)
>>
>>
>> On Tue, Jun 5, 2012 at 3:44 AM, Keith Wiley <[EMAIL PROTECTED]> wrote:
>>>
>>> The number of slots corresponds to the number of cores (e.g., a 10-node
>>> cluster consisting of quad-core cpus has 40 cores and therefore 40 slots).
>>>  The slots can be divided amongst the maps and reduces in a way which
>>> integer-divides into the cores of a single machine.  So if the nodes are
>>> quad-cores you can do 3:1, 2:2, or 1:3.  If the nodes are dual-quad-cores
>>> (each node has eight cores) you can do 7:1 through 1:7 in any configuration
>>> you want..........as per my understanding.  Others may pipe in to clarify
>>> some minor point I have missed, I'm unsure.
>>>
>>> On Jun 4, 2012, at 09:06 , kartheek muthyala wrote:
>>>
>>> > Hello all,
>>> >
>>> > Can I increase the number of map slots in a machine using some
>>> > configuration?. I can find configurations for increasing the number of
>>> > maps and reduce tasks that can run, but didn't find anything related
>>> > to slots. Is that default decided based on the memory size?
>>> >
>>> > Thanks,
>>> > Kartheek.
>>>
>>>
>>>
>>> ________________________________________________________________________________
>>> Keith Wiley     [EMAIL PROTECTED]     keithwiley.com
>>>  music.keithwiley.com
>>>
>>> "I do not feel obliged to believe that the same God who has endowed us
>>> with
>>> sense, reason, and intellect has intended us to forgo their use."
>>>                                           --  Galileo Galilei
>>>
>>> ________________________________________________________________________________
>>>
>>
>>
>>
>> --
>> https://github.com/zinnia-phatak-dev/Nectar
>>

--
Harsh J