Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
MapReduce >> mail # user >> Memory based scheduling


+
Marco Zühlke 2012-10-30, 15:49
+
Arun C Murthy 2012-10-30, 17:24
+
lohit 2012-10-30, 16:08
+
Arun C Murthy 2012-10-30, 17:24
+
Harsh J 2012-10-30, 17:50
Copy link to this message
-
Re: Memory based scheduling
I did not know about this feature in Capacity Scheduler.
Just to get more information, lets assume cluster has 4 slots on node with
8G (as mentioned by Marco). Now if 2 slots are occupied with 4G memory,
what happens if another job which gets scheduled which asks for only 2G per
slot. Does it get scheduled? if so, does it get killed instead of the 4G
slots?
2012/10/30 Arun C Murthy <[EMAIL PROTECTED]>

> Not true, take a look at my prev. response.
>
> On Oct 30, 2012, at 9:08 AM, lohit wrote:
>
> As far as I recall this is not possible. Per job or per user
> configurations like these are little difficult in existing version.
> What you could try is to set max map per job to be say half of cluster
> capacity. (This is possible with FairSchedule, I do not know of
> CapacityScheduler)
> For eg, if you have 10 nodes with 4 slots each. You would create pool and
> set max maps to be 20.
> JobTracker will try its best to spread tasks across nodes provided they
> are empty slots. But again, this is not guaranteed.
>
>
> 2012/10/30 Marco Zühlke <[EMAIL PROTECTED]>
>
>> Hi,
>>
>> on our cluster our jobs usually satisfied with less than 2 GB of heap
>> space.
>> so we have on our 8 GB computers 3 maps maximum and on our 16 GB
>> computers 4 maps maximum (we only have quad core CPUs and to have
>> memory left for reducers). This works very well.
>>
>> But now we have a new kind of jobs. Each mapper requires at lest 4 GB
>> of heap space.
>>
>> Is it possible to limit the number of tasks (mapper) per computer to 1 or
>> 2 for
>> these kinds of jobs ?
>>
>> Regards,
>> Marco
>>
>>
>
>
> --
> Have a Nice Day!
> Lohit
>
>
> --
> Arun C. Murthy
> Hortonworks Inc.
> http://hortonworks.com/
>
>
>
--
Have a Nice Day!
Lohit
+
Kaczmarek, Eric 2012-10-30, 15:58