Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce >> mail # user >> Re: Allocating Containers on a particular Node in Yarn


Copy link to this message
-
Re: Allocating Containers on a particular Node in Yarn
Great to hear.  Other answers inline
On Thu, Nov 14, 2013 at 12:05 PM, Gaurav Gupta <[EMAIL PROTECTED]>wrote:

> Sandy,
>
>
>
> The last trick worked but now I have couple of more questions
>
>
>
> 1.      If I don’t request for rack and relax locality is false with
> scheduler delay on , I see that when I pass a wrong host I don’t get any
> container back. Why so?
>
When locality relaxation is turned on, we will only ever place the
container on the host that you requested.  This means that if no such host
exists, we will never place the container.  This behavior could maybe be
improved, but there are some tricky issues to deal with there about nodes
going down and such.

> 2.      I also noticed that with scheduler delay on and relax locality
> set to false, If I request for rack along with a host I get one container
> on expected node but not other containers.
>
> 3.      With scheduler delay on and relax locality set to true without
> requesting rack, I don’t get the containers on the required host
>
What scheduler are you using and what properties are you using to turn the
scheduler delay on?

>
>
> Thanks
>
> -Gaurav
>
>
>
> *From:* Sandy Ryza [mailto:[EMAIL PROTECTED]]
> *Sent:* Thursday, November 14, 2013 11:41 AM
> *To:* [EMAIL PROTECTED]
>
> *Subject:* Re: Allocating Containers on a particular Node in Yarn
>
>
>
> Requesting the rack is not necessary, and is leading to the behavior that
> you're seeing.
>
>
>
> The documentation states:
>
>    * <li>If locality relaxation is disabled, then only within the same
> request,
>
>    * a node and its rack may be specified together. This allows for a
> specific
>
>    * rack with a preference for a specific node within that rack.</li>
>
>
>
>
>
> On Wed, Nov 13, 2013 at 10:10 PM, gaurav <[EMAIL PROTECTED]> wrote:
>
> Here is the snippet of code that I am using to allocate containers
>
>     AMRMClient<ContainerRequest> amRmClient > AMRMClient.createAMRMClient();;
>     String host = "h1";
>     Resource capability = Records.newRecord(Resource.class);
>     capability.setMemory(memory);
>     nodes = new String[] {host};
>     // in order to request a host, we also have to request the rack
>     racks = new String[] {"/default-rack"};
>      List<ContainerRequest> containerRequests = new
> ArrayList<ContainerRequest>();
>     List<ContainerId> releasedContainers = new ArrayList<ContainerId>();
>     containerRequests.add(new ContainerRequest(capability, nodes, racks,
> Priority.newInstance(priority),false));
>     if (containerRequests.size() > 0) {
>       LOG.info("Asking RM for containers: " + containerRequests);
>       for (ContainerRequest cr : containerRequests) {
>         LOG.info("Requested container: {}", cr.toString());
>         amRmClient.addContainerRequest(cr);
>       }
>     }
>
>     for (ContainerId containerId : releasedContainers) {
>       LOG.info("Released container, id={}", containerId.getId());
>       amRmClient.releaseAssignedContainer(containerId);
>     }
>     return amRmClient.allocate(0);
>
>
>
> -Gaurav
>
>
>
> On 11/13/2013 07:36 PM, Sandy Ryza wrote:
>
> In that case, the AMRMClient code looks correct to me.  Can you share the
> code you've written against it that's not receiving the correct containers?
>
>
>
> On Wed, Nov 13, 2013 at 5:30 PM, gaurav <[EMAIL PROTECTED]> wrote:
>
> Hi,
>
> I looked in the trunk and line numbers are 361 and 366.
>
> Thanks
>
> -Gaurav
>
> On 11/13/2013 5:04 PM, gaurav wrote:
>
> I have hadoop-2.2.0
>
> Thanks
>
> -Gaurav
>
> On 11/13/2013 4:59 PM, Sandy Ryza wrote:
>
> What version are you using?  Setting the relax locality to true for nodes
> is always correct.  For racks, this is not necessarily the case.  When I
> look at trunk, it is set to true always on line 361 (which is correct), but
> on on line 374.
>
>
>
> -Sandy
>
>
>
> On Wed, Nov 13, 2013 at 4:47 PM, gaurav <[EMAIL PROTECTED]> wrote:
>
> Hi Sandy,
>
> No it is not working for me.  As mentioned earlier, AMRMClient is not
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB