Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
HDFS >> mail # user >> Increase number of map slots


+
kartheek muthyala 2012-06-04, 16:06
+
Keith Wiley 2012-06-04, 22:14
+
madhu phatak 2012-06-05, 05:35
+
kartheek muthyala 2012-06-06, 04:26
Copy link to this message
-
Re: Increase number of map slots
> I think mapred.tasktracker.map.tasks.maximum sets the number of map
tasks and not slots.

This is incorrect. The property does configure slots. Please also see
http://wiki.apache.org/hadoop/HowManyMapsAndReduces and
http://wiki.apache.org/hadoop/FAQ#I_see_a_maximum_of_2_maps.2BAC8-reduces_spawned_concurrently_on_each_TaskTracker.2C_how_do_I_increase_that.3F
for more.

On Wed, Jun 6, 2012 at 9:56 AM, kartheek muthyala
<[EMAIL PROTECTED]> wrote:
> I think mapred.tasktracker.map.tasks.maximum sets the number of map
> tasks and not slots.
> Keith, so that means per core, if I specify the maximum number of map
> tasks it can host, whether all the maps per slot will run parallely or
> they execute one after the other?. I understand that maps that are
> scheduled on different cores can run parallely.
>
> Thanks,
> Kartheek.
>
> On Tue, Jun 5, 2012 at 11:05 AM, madhu phatak <[EMAIL PROTECTED]> wrote:
>> Hi,
>>  You can change the no of slots using the following properties in
>> mapred-site.xml.
>>
>>  mapred.tasktracker.map.tasks.maximum ( for map slots)
>>  mapred.tasktracker.reduce.tasks.maximum (for reduce slot)
>>
>>
>> On Tue, Jun 5, 2012 at 3:44 AM, Keith Wiley <[EMAIL PROTECTED]> wrote:
>>>
>>> The number of slots corresponds to the number of cores (e.g., a 10-node
>>> cluster consisting of quad-core cpus has 40 cores and therefore 40 slots).
>>>  The slots can be divided amongst the maps and reduces in a way which
>>> integer-divides into the cores of a single machine.  So if the nodes are
>>> quad-cores you can do 3:1, 2:2, or 1:3.  If the nodes are dual-quad-cores
>>> (each node has eight cores) you can do 7:1 through 1:7 in any configuration
>>> you want..........as per my understanding.  Others may pipe in to clarify
>>> some minor point I have missed, I'm unsure.
>>>
>>> On Jun 4, 2012, at 09:06 , kartheek muthyala wrote:
>>>
>>> > Hello all,
>>> >
>>> > Can I increase the number of map slots in a machine using some
>>> > configuration?. I can find configurations for increasing the number of
>>> > maps and reduce tasks that can run, but didn't find anything related
>>> > to slots. Is that default decided based on the memory size?
>>> >
>>> > Thanks,
>>> > Kartheek.
>>>
>>>
>>>
>>> ________________________________________________________________________________
>>> Keith Wiley     [EMAIL PROTECTED]     keithwiley.com
>>>  music.keithwiley.com
>>>
>>> "I do not feel obliged to believe that the same God who has endowed us
>>> with
>>> sense, reason, and intellect has intended us to forgo their use."
>>>                                           --  Galileo Galilei
>>>
>>> ________________________________________________________________________________
>>>
>>
>>
>>
>> --
>> https://github.com/zinnia-phatak-dev/Nectar
>>

--
Harsh J
+
Keith Wiley 2012-06-06, 17:08
+
Harsh J 2012-06-14, 15:11
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB