-Re: Capacity Scheduler question
Serge Blazhievsky 2013-03-22, 22:06
Take a look at fair scheduler it will do what you ask for
Sent from my iPhone
On Mar 22, 2013, at 2:48 PM, jeremy p <[EMAIL PROTECTED]> wrote:
> I have two jobs, Job A and Job B. Job A needs to run with 18 mappers per machine, Job B needs to run with 1 mapper per machine. Hadoop doesn't give you a way to specify number of mappers on a per-job basis. mapred.tasktracker.map.tasks.maximum and mapred.map.tasks do absolutely nothing. I've been looking into the Capacity Scheduler, but I'm unsure if it can help me. In this documentation, all the settings under "Resource Allocation" are cluster-wide. I need to be able to set the maximum capacity on a given machine. It does look like you have the option to set the required amount of memory per slot, but that setting applies to all the queues. If I could set that value on a per-queue basis, that would be helpful.
> Will the capacity scheduler help me here? Or am I barking up the wrong tree? If the capacity scheduler won't help me, can you think of anything that will?