Its possible to configure in YARN scheduling requirement to not run more
than 1 mapper per node?

Its not because of memory requirements. Who decides this kind of
scheduling? ResourceManager or ApplicationManager?

I do not believe that there is currently a way to do this.

Do you want it so that the node can run only 1 mapper at a time but the node can also run multiple reducers at the same time?  Do you want it so that this particular application will only use that node for a single mapper, but other applications may also run mappers on the node?

The RM has no knowledge about mapper vs. reducer, but the AM has no knowledge about other applications so it may require modifications in multiple places.

--Bobby Evans

On 4/26/12 12:56 PM, "Radim Kolar" <[EMAIL PROTECTED]> wrote:

Its possible to configure in YARN scheduling requirement to not run more
than 1 mapper per node?

Its not because of memory requirements. Who decides this kind of
scheduling? ResourceManager or ApplicationManager?

> Do you want it so that the node can run only 1 mapper at a time but
> the node can also run multiple reducers at the same time?
I currently want it per-application. Node can run more mappers/reducers
from other jobs at same time.
>  Do you want it so that this particular application will only use that
> node for a single mapper, but other applications may also run mappers
> on the node?
yes. After this will be coded, i will rerun tests if it is sufficient
for other types of tasks too.

For start, i would need per job settings - max mappers/reducers per
node, so modifying AM would be enough.

Radim,

You would want to modify the application master for this, and it is likely to be a bit of a hack because the RM scheduler itself is not really designed for this.  You can have the AM return containers to the RM that it does not want (which it does already in some cases), because they are on the same node as an already running map task.  This could slow down your application a lot, as it makes it more difficult for the RM to find free containers that the AM wants.  You could also modify the AM so that it will not request any more then just one container per node, which would help with this, but not solve it completely.

--Bobby Evans

On 4/27/12 12:38 AM, "Radim Kolar" <[EMAIL PROTECTED]> wrote:

Do you want it so that the node can run only 1 mapper at a time but the node can also run multiple reducers at the same time?
I currently want it per-application. Node can run more mappers/reducers from other jobs at same time.

 Do you want it so that this particular application will only use that node for a single mapper, but other applications may also run mappers on the node?
yes. After this will be coded, i will rerun tests if it is sufficient for other types of tasks too.

 For start, i would need per job settings - max mappers/reducers per node, so modifying AM would be enough.

Dne 27.4.2012 17:30, Robert Evans napsal(a):
> Radim,
>
> You would want to modify the application master for this, and it is
> likely to be a bit of a hack because the RM scheduler itself is not
> really designed for this.
What about to do something like this:
  In job JAR there will be loadable plugin which will be loaded by AM
and used for allocating containers. For example mapreduce job should
request to launch only data-local tasks or do some other customization.

if plugin system for AM is overkill, something simpler can be made like:

maximum number of mappers per node
maximum number of reducers per node

maximum percentage of non data local tasks
maximum percentage of rack local tasks

and set this in job properties.

Either plugins or configuration options would be possible to do.  The real question is what is the use case for this, and is that use case big enough to warrant it being part of core Hadoop?  I have seen a few situations where this perhaps makes since, but most of those are because the resource scheduling is currently very basic (meaning it only knows abut RAM and machine/rack locality which is really a surrogate for requesting a HDFS block as a resource).  For example GPGPU computation, where sharing the GPU between processes can be complicated, or more commonly where the network is the bottleneck and multiple processes running on the same box is not an optimal solution because it will saturate the network.

In both of those cases I would prefer to see resource scheduling updated rather then trying to work around it by having the AM throw away containers.  But that is just me and is a major undertaking.  Throwing away the containers feels like a hack to me but it is something that can work right now.  It just depends on what the use case is and if it is compelling enough to make that hack a fully supported part of the Hadoop map/reduce API.

--Bobby Evans
On 5/3/12 7:59 AM, "Radim Kolar" <[EMAIL PROTECTED]> wrote:

if plugin system for AM is overkill, something simpler can be made like:

maximum number of mappers per node
maximum number of reducers per node

maximum percentage of non data local tasks
maximum percentage of rack local tasks

and set this in job properties.

We've been against these 'features' since it leads to very bad behaviour across the cluster with multiple apps/users etc.

What is your use-case i.e. what are you trying to achieve with this?

thanks,
Arun

On May 3, 2012, at 5:59 AM, Radim Kolar wrote:

> if plugin system for AM is overkill, something simpler can be made like:
>
> maximum number of mappers per node
> maximum number of reducers per node
>
> maximum percentage of non data local tasks
> maximum percentage of rack local tasks
>
> and set this in job properties.

--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/

Not to speak for Radim, but what I'm trying to achieve is performance at least as good as 0.20 for all cases.  That is, no regressions.  For something as simple as terasort, I don't think that is possible without being able to specify the max number of mappers/reducers per node.  As it is, I see slowdowns as much as 2X.  Hopefully I'm wrong and somebody will straighten me out.  But if I'm not, adding such a feature won't lead to bad behavior of any kind since the default could be set to unlimited and thus have no effect whatsoever.

I should emphasize that I support the goal of greater automation since Hadoop has way too many parameters and is so hard to tune.  Just not at the expense of performance regressions.

Jeff
We've been against these 'features' since it leads to very bad behaviour across the cluster with multiple apps/users etc.

What is your use-case i.e. what are you trying to achieve with this?

thanks,
Arun

On May 3, 2012, at 5:59 AM, Radim Kolar wrote:
if plugin system for AM is overkill, something simpler can be made like:

maximum number of mappers per node
maximum number of reducers per node

maximum percentage of non data local tasks
maximum percentage of rack local tasks

and set this in job properties.

> We've been against these 'features' since it leads to very bad
> behaviour across the cluster with multiple apps/users etc.
Its not new feature, its extension of existing resource scheduling which
works good enough only for RAM. There are 2 other resources - CPU cores
and network IO which needs to be considered.

We have job which is doing lot of network IO in mapper and its desirable
to run mappers on different nodes even if reading blocks from HDFS will
not be local.

Our second job is burning all CPU cores on machine while doing
computations, its important for mappers not to land on same node.

One nice feature is to attach different configuration for different job.
You can do it either by command line "-conf" or programatically.  We are
using that extensively while experimenting different setting.

Zhu, Guojun
Modeling Sr Graduate
571-3824370
[EMAIL PROTECTED]
Financial Engineering
Freddie Mac

   Radim Kolar <[EMAIL PROTECTED]>
   05/10/2012 07:56 AM
   Please respond to
[EMAIL PROTECTED]
To
[EMAIL PROTECTED]
cc

Subject
Re: max 1 mapper per node

> We've been against these 'features' since it leads to very bad
> behaviour across the cluster with multiple apps/users etc.
Its not new feature, its extension of existing resource scheduling which
works good enough only for RAM. There are 2 other resources - CPU cores
and network IO which needs to be considered.

We have job which is doing lot of network IO in mapper and its desirable
to run mappers on different nodes even if reading blocks from HDFS will
not be local.

Our second job is burning all CPU cores on machine while doing
computations, its important for mappers not to land on same node.

Yes adding in more resources in the scheduling request would be the ideal solution to the problem.  But sadly that is not a trivial change.  The initial solution I suggested is an ugly hack, and will not work for the cases you have suggested.  If you feel that this is important work please feel free to file a JIRA for this.  We can continue discussion on that JIRA about  the details of how to add in this type of functionality.  I am very interested in the scheduler and would be happy to help out, but sadly my time right now is very limited.

--Bobby Evans

On 5/10/12 6:56 AM, "Radim Kolar" <[EMAIL PROTECTED]> wrote:

> We've been against these 'features' since it leads to very bad
> behaviour across the cluster with multiple apps/users etc.
Its not new feature, its extension of existing resource scheduling which
works good enough only for RAM. There are 2 other resources - CPU cores
and network IO which needs to be considered.

We have job which is doing lot of network IO in mapper and its desirable
to run mappers on different nodes even if reading blocks from HDFS will
not be local.

Our second job is burning all CPU cores on machine while doing
computations, its important for mappers not to land on same node.

For terasort you want to fill up your entire cluster with maps/reduces as fast as you can to get the best performance.

Just play with #slots.

Arun

On May 9, 2012, at 12:36 PM, Jeffrey Buell wrote:

> Not to speak for Radim, but what I’m trying to achieve is performance at least as good as 0.20 for all cases.  That is, no regressions.  For something as simple as terasort, I don’t think that is possible without being able to specify the max number of mappers/reducers per node.  As it is, I see slowdowns as much as 2X.  Hopefully I’m wrong and somebody will straighten me out.  But if I’m not, adding such a feature won’t lead to bad behavior of any kind since the default could be set to unlimited and thus have no effect whatsoever.
>  
> I should emphasize that I support the goal of greater automation since Hadoop has way too many parameters and is so hard to tune.  Just not at the expense of performance regressions.
>  
> Jeff
>  
>  
> We've been against these 'features' since it leads to very bad behaviour across the cluster with multiple apps/users etc.
>  
> What is your use-case i.e. what are you trying to achieve with this?
>  
> thanks,
> Arun
>  
> On May 3, 2012, at 5:59 AM, Radim Kolar wrote:
>
>
> if plugin system for AM is overkill, something simpler can be made like:
>
> maximum number of mappers per node
> maximum number of reducers per node
>
> maximum percentage of non data local tasks
> maximum percentage of rack local tasks
>
> and set this in job properties.
>  
>  

--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/

I have the right #slots to fill up memory across the cluster, and all those slots are filled with tasks. The problem I ran into was that the maps grabbed all the slots initially and the reduces had a hard time getting started.  As maps finished, more maps were started and only rarely was a reduce started.  I assume this behavior occurred because I had ~4000 map tasks in the queue, but only ~100 reduce tasks.  If the scheduler lumps maps and reduces together, then whenever a slot opens up it will almost surely be taken by a map task.  To get good performance I need all reduce tasks started early on, and have only map tasks compete for open slots.  Other apps may need different priorities between maps and reduces.  In any case, I don't understand how treating maps and reduces the same is workable.

Jeff

From: Arun C Murthy [mailto:[EMAIL PROTECTED]]
Sent: Thursday, May 10, 2012 1:27 PM
To: [EMAIL PROTECTED]
Subject: Re: max 1 mapper per node

For terasort you want to fill up your entire cluster with maps/reduces as fast as you can to get the best performance.

Just play with #slots.

Arun

On May 9, 2012, at 12:36 PM, Jeffrey Buell wrote:
Not to speak for Radim, but what I'm trying to achieve is performance at least as good as 0.20 for all cases.  That is, no regressions.  For something as simple as terasort, I don't think that is possible without being able to specify the max number of mappers/reducers per node.  As it is, I see slowdowns as much as 2X.  Hopefully I'm wrong and somebody will straighten me out.  But if I'm not, adding such a feature won't lead to bad behavior of any kind since the default could be set to unlimited and thus have no effect whatsoever.

I should emphasize that I support the goal of greater automation since Hadoop has way too many parameters and is so hard to tune.  Just not at the expense of performance regressions.

Jeff
We've been against these 'features' since it leads to very bad behaviour across the cluster with multiple apps/users etc.

What is your use-case i.e. what are you trying to achieve with this?

thanks,
Arun

On May 3, 2012, at 5:59 AM, Radim Kolar wrote:

if plugin system for AM is overkill, something simpler can be made like:

maximum number of mappers per node
maximum number of reducers per node

maximum percentage of non data local tasks
maximum percentage of rack local tasks

and set this in job properties.

--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/

Dne 10.5.2012 15:29, Robert Evans napsal(a):
> Yes adding in more resources in the scheduling request would be the
> ideal solution to the problem.  But sadly that is not a trivial change.
Best will be to have custom resources.

Example: in node config define that this node has:

1 x GPU
1 x Gbit Network
etc.

then add property to job configuration that this job needs 1 GPU. RM
will pickup node with free GPU.