Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
MapReduce, mail # user - Re: Allocating Containers on a particular Node in Yarn


+
gaurav 2013-11-14, 00:47
+
Sandy Ryza 2013-11-14, 00:59
+
gaurav 2013-11-14, 01:04
+
gaurav 2013-11-14, 01:30
+
Sandy Ryza 2013-11-14, 03:36
+
gaurav 2013-11-14, 06:10
+
Sandy Ryza 2013-11-14, 19:41
+
Gaurav Gupta 2013-11-14, 20:05
+
Sandy Ryza 2013-11-14, 20:40
+
Gaurav Gupta 2013-11-14, 21:28
Copy link to this message
-
Re: Allocating Containers on a particular Node in Yarn
Sandy Ryza 2013-11-14, 21:33
Yeah, you're right, I meant when locality relaxation is turned off.

I'm not super familiar with the capacity scheduler (more familiar with the
fair scheduler), so maybe someone with knowledge about that can chime in.
On Thu, Nov 14, 2013 at 1:28 PM, Gaurav Gupta <[EMAIL PROTECTED]>wrote:

> Sandy,
>
>
>
> For the first question you mentioned relaxation locality is turn on, I
> assume you mean that relax locality is set to false right?
>
>
>
> For scheduler, I am using capacity scheduler and modified following
> property in capacity-scheduler.xml
>
> <property>
>
> <name>yarn.scheduler.capacity.node-locality-delay</name>
>
>      <value>1</value>
>
>      <description>
>
>        Number of missed scheduling opportunities after which the
> CapacityScheduler
>
>        attempts to schedule rack-local containers.
>
>        Typically this should be set to number of racks in the cluster, this
>
>        feature is disabled by default, set to -1.
>
>      </description>
>
>    </property>
>
>
>
> -Gaurav
>
>
>
> *From:* Sandy Ryza [mailto:[EMAIL PROTECTED]]
> *Sent:* Thursday, November 14, 2013 12:41 PM
>
> *To:* [EMAIL PROTECTED]
> *Subject:* Re: Allocating Containers on a particular Node in Yarn
>
>
>
> Great to hear.  Other answers inline
>
>
>
> On Thu, Nov 14, 2013 at 12:05 PM, Gaurav Gupta <[EMAIL PROTECTED]>
> wrote:
>
> Sandy,
>
>
>
> The last trick worked but now I have couple of more questions
>
>
>
> 1.      If I don’t request for rack and relax locality is false with
> scheduler delay on , I see that when I pass a wrong host I don’t get any
> container back. Why so?
>
> When locality relaxation is turned on, we will only ever place the
> container on the host that you requested.  This means that if no such host
> exists, we will never place the container.  This behavior could maybe be
> improved, but there are some tricky issues to deal with there about nodes
> going down and such.
>
> 2.      I also noticed that with scheduler delay on and relax locality
> set to false, If I request for rack along with a host I get one container
> on expected node but not other containers.
>
> 3.      With scheduler delay on and relax locality set to true without
> requesting rack, I don’t get the containers on the required host
>
> What scheduler are you using and what properties are you using to turn the
> scheduler delay on?
>
>
>
> Thanks
>
> -Gaurav
>
>
>
> *From:* Sandy Ryza [mailto:[EMAIL PROTECTED]]
> *Sent:* Thursday, November 14, 2013 11:41 AM
> *To:* [EMAIL PROTECTED]
>
>
> *Subject:* Re: Allocating Containers on a particular Node in Yarn
>
>
>
> Requesting the rack is not necessary, and is leading to the behavior that
> you're seeing.
>
>
>
> The documentation states:
>
>    * <li>If locality relaxation is disabled, then only within the same
> request,
>
>    * a node and its rack may be specified together. This allows for a
> specific
>
>    * rack with a preference for a specific node within that rack.</li>
>
>
>
>
>
> On Wed, Nov 13, 2013 at 10:10 PM, gaurav <[EMAIL PROTECTED]> wrote:
>
> Here is the snippet of code that I am using to allocate containers
>
>     AMRMClient<ContainerRequest> amRmClient > AMRMClient.createAMRMClient();;
>     String host = "h1";
>     Resource capability = Records.newRecord(Resource.class);
>     capability.setMemory(memory);
>     nodes = new String[] {host};
>     // in order to request a host, we also have to request the rack
>     racks = new String[] {"/default-rack"};
>      List<ContainerRequest> containerRequests = new
> ArrayList<ContainerRequest>();
>     List<ContainerId> releasedContainers = new ArrayList<ContainerId>();
>     containerRequests.add(new ContainerRequest(capability, nodes, racks,
> Priority.newInstance(priority),false));
>     if (containerRequests.size() > 0) {
>       LOG.info("Asking RM for containers: " + containerRequests);
>       for (ContainerRequest cr : containerRequests) {
>         LOG.info("Requested container: {}", cr.toString());