Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HDFS >> mail # user >> container allocation


Copy link to this message
-
RE: container allocation
Thanks that is good to know.  Is there any way to say "please fail if I don't get the node I want?"  Do I just release the container and try again?
I'd like to understand the implications of this policy.  Suppose I have 1000 data splits and cluster capacity of 100 containers.  If I try to schedule 200 tasks, requesting a local data node for each one, how do I ensure the highest chance that the tasks run against local data?  Do I just ask for all 200 at once?  Should I ask for 100 at a time and then re-target the remainder as containers come open?
Or am I thinking about this all wrong... perhaps I should ask for containers, see what nodes they are on, and then assign the data splits to them once I see the set of available containers?
john
From: Arun C Murthy [mailto:[EMAIL PROTECTED]]
Sent: Thursday, June 13, 2013 12:27 AM
To: [EMAIL PROTECTED]
Subject: Re: container allocation

By default, the ResourceManager will try give you a container on that node, rack or anywhere (in that order).

We recently added ability to whitelist or blacklist nodes to allow for more control.

Arun

On Jun 12, 2013, at 8:03 AM, John Lilley wrote:
If I request a container on a node, and that node is busy, will the request fail, or will it give me a container on a different node?  In other words is the node name a requirement or a hint?
Thanks
John
--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/

NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB