Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce >> mail # user >> Capacity Scheduler question


Copy link to this message
-
Re: Capacity Scheduler question
Take a look at fair scheduler it will do what you ask for

Sent from my iPhone

On Mar 22, 2013, at 2:48 PM, jeremy p <[EMAIL PROTECTED]> wrote:

> I have two jobs, Job A and Job B.  Job A needs to run with 18 mappers per machine, Job B needs to run with 1 mapper per machine.  Hadoop doesn't give you a way to specify number of mappers on a per-job basis.  mapred.tasktracker.map.tasks.maximum and mapred.map.tasks do absolutely nothing.  I've been looking into the Capacity Scheduler, but I'm unsure if it can help me.  In this documentation, all the settings under "Resource Allocation" are cluster-wide.  I need to be able to set the maximum capacity on a given machine.  It does look like you have the option to set the required amount of memory per slot, but that setting applies to all the queues.  If I could set that value on a per-queue basis, that would be helpful.
>
> Will the capacity scheduler help me here?  Or am I barking up the wrong tree?  If the capacity scheduler won't help me, can you think of anything that will?
>
> Thanks!
>
> --Jeremy