Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
HDFS >> mail # user >> container allocation


+
John Lilley 2013-06-12, 15:03
+
Arun C Murthy 2013-06-13, 06:26
+
John Lilley 2013-06-14, 19:41
Copy link to this message
-
Re: container allocation
Hi John,

At this time, releasing containers is the preferred way to be strict about
your locality requirements.  This is not included in a release yet, but
https://issues.apache.org/jira/browse/YARN-392 allows expressing hard
locality constraints on requests, so you can tell the scheduler to never
give you a node that you didn't ask for.

-Sandy
On Fri, Jun 14, 2013 at 12:41 PM, John Lilley <[EMAIL PROTECTED]>wrote:

>  Thanks that is good to know.  Is there any way to say “please fail if I
> don’t get the node I want?”  Do I just release the container and try
> again?  ****
>
> I’d like to understand the implications of this policy.  Suppose I have
> 1000 data splits and cluster capacity of 100 containers.  If I try to
> schedule 200 tasks, requesting a local data node for each one, how do I
> ensure the highest chance that the tasks run against local data?  Do I just
> ask for all 200 at once?  Should I ask for 100 at a time and then re-target
> the remainder as containers come open?****
>
> Or am I thinking about this all wrong… perhaps I should ask for
> containers, see what nodes they are on, and then assign the data splits to
> them once I see the set of available containers?****
>
> john****
>
> *From:* Arun C Murthy [mailto:[EMAIL PROTECTED]]
> *Sent:* Thursday, June 13, 2013 12:27 AM
> *To:* [EMAIL PROTECTED]
> *Subject:* Re: container allocation****
>
> ** **
>
> By default, the ResourceManager will try give you a container on that
> node, rack or anywhere (in that order).****
>
> ** **
>
> We recently added ability to whitelist or blacklist nodes to allow for
> more control.****
>
> ** **
>
> Arun****
>
> ** **
>
> On Jun 12, 2013, at 8:03 AM, John Lilley wrote:****
>
>
>
> ****
>
> If I request a container on a node, and that node is busy, will the
> request fail, or will it give me a container on a different node?  In other
> words is the node name a requirement or a hint?****
>
> Thanks****
>
> John****
>
>  ****
>
> ** **
>
> --****
>
> Arun C. Murthy****
>
> Hortonworks Inc.
> http://hortonworks.com/****
>
> ** **
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB