Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
MapReduce >> mail # user >> Memory based scheduling


+
Marco Zühlke 2012-10-30, 15:49
Copy link to this message
-
Re: Memory based scheduling
Yes, use the CapacityScheduler and ask for multiple slots per map or reduce task:
http://hadoop.apache.org/docs/stable/capacity_scheduler.html#Resource+based+scheduling

Arun

On Oct 30, 2012, at 8:49 AM, Marco Zühlke wrote:

> Hi,
>
> on our cluster our jobs usually satisfied with less than 2 GB of heap space.
> so we have on our 8 GB computers 3 maps maximum and on our 16 GB
> computers 4 maps maximum (we only have quad core CPUs and to have
> memory left for reducers). This works very well.
>
> But now we have a new kind of jobs. Each mapper requires at lest 4 GB
> of heap space.
>
> Is it possible to limit the number of tasks (mapper) per computer to 1 or 2 for
> these kinds of jobs ?
>
> Regards,
> Marco
>

--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/
+
lohit 2012-10-30, 16:08
+
Arun C Murthy 2012-10-30, 17:24
+
Harsh J 2012-10-30, 17:50
+
lohit 2012-10-30, 17:33
+
Kaczmarek, Eric 2012-10-30, 15:58