-Re: Changing the maximum tasks per node on a per job basis
Harsh J 2013-05-23, 15:37
Your problem seems to surround available memory and over-subscription. If
you're using a 0.20.x or 1.x version of Apache Hadoop, you probably want to
use the CapacityScheduler to address this for you.
I once detailed how-to, on a similar question here:
On Wed, May 22, 2013 at 2:55 PM, Steve Lewis <[EMAIL PROTECTED]> wrote:
> I have a series of Hadoop jobs to run - one of my jobs requires larger than
> standard memory
> I allow the task to use 2GB of memory. When I run some of these jobs the
> slave nodes are crashing because they run out of swap space. It is not that
> s slave count not run one. or even 4 of these jobs but 8 stresses the
> I could cut the mapred.tasktracker.reduce.tasks.maximum for the entire
> cluster but this cripples the whole cluster for one of many jobs.
> It seems to be a very bad design
> a) to allow the job tracker to keep assigning tasks to a slave that is
> already getting low on memory
> b) to allow the user to run jobs capable or crashing noeds on the cluster
> c) not to allow the user to specify that some jobs need to be limited to a
> lower value without requiring this limit for every job.
> Are there plans to fix this??