Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce >> mail # user >> Re: Allocating Containers on a particular Node in Yarn


Copy link to this message
-
Re: Allocating Containers on a particular Node in Yarn
Requesting the rack is not necessary, and is leading to the behavior that
you're seeing.

The documentation states:
   * <li>If locality relaxation is disabled, then only within the same
request,
   * a node and its rack may be specified together. This allows for a
specific
   * rack with a preference for a specific node within that rack.</li>

On Wed, Nov 13, 2013 at 10:10 PM, gaurav <[EMAIL PROTECTED]> wrote:

>  Here is the snippet of code that I am using to allocate containers
>
>     AMRMClient<ContainerRequest> amRmClient > AMRMClient.createAMRMClient();;
>     String host = "h1";
>     Resource capability = Records.newRecord(Resource.class);
>     capability.setMemory(memory);
>     nodes = new String[] {host};
>     // in order to request a host, we also have to request the rack
>     racks = new String[] {"/default-rack"};
>      List<ContainerRequest> containerRequests = new
> ArrayList<ContainerRequest>();
>     List<ContainerId> releasedContainers = new ArrayList<ContainerId>();
>     containerRequests.add(new ContainerRequest(capability, nodes, racks,
> Priority.newInstance(priority),false));
>     if (containerRequests.size() > 0) {
>       LOG.info("Asking RM for containers: " + containerRequests);
>       for (ContainerRequest cr : containerRequests) {
>         LOG.info("Requested container: {}", cr.toString());
>         amRmClient.addContainerRequest(cr);
>       }
>     }
>
>     for (ContainerId containerId : releasedContainers) {
>       LOG.info("Released container, id={}", containerId.getId());
>       amRmClient.releaseAssignedContainer(containerId);
>     }
>     return amRmClient.allocate(0);
>
>
>
> -Gaurav
>
>
> On 11/13/2013 07:36 PM, Sandy Ryza wrote:
>
> In that case, the AMRMClient code looks correct to me.  Can you share the
> code you've written against it that's not receiving the correct containers?
>
>
>  On Wed, Nov 13, 2013 at 5:30 PM, gaurav <[EMAIL PROTECTED]> wrote:
>
>>  Hi,
>>
>> I looked in the trunk and line numbers are 361 and 366.
>>
>> Thanks
>> -Gaurav
>>
>>  On 11/13/2013 5:04 PM, gaurav wrote:
>>
>> I have hadoop-2.2.0
>>
>> Thanks
>> -Gaurav
>>
>> On 11/13/2013 4:59 PM, Sandy Ryza wrote:
>>
>> What version are you using?  Setting the relax locality to true for nodes
>> is always correct.  For racks, this is not necessarily the case.  When I
>> look at trunk, it is set to true always on line 361 (which is correct), but
>> on on line 374.
>>
>>  -Sandy
>>
>>
>> On Wed, Nov 13, 2013 at 4:47 PM, gaurav <[EMAIL PROTECTED]> wrote:
>>
>>>  Hi Sandy,
>>>
>>> No it is not working for me.  As mentioned earlier, AMRMClient is not
>>> respecting the locality for node and rack in *Line 361 and 374 *and is
>>> set to true always.
>>> I am requesting for one container with specified node and rack and
>>> relaxed locality false.
>>>
>>> Thanks
>>> -Gaurav
>>>
>>>  On 11/13/2013 4:24 PM, Sandy Ryza wrote:
>>>
>>>  [moving to user list]
>>>
>>>  Right.  relaxLocality needs to be set on the next level up.  It
>>> determines whether locality can be relaxed to that level.  Confusing, I
>>> know.  If you are using AMRMClient, you should be able to accomplish what
>>> you're looking for by creating an AMRMClient.ContainerRequest that
>>> specifies a node and sets relax locality to false.  Is that not working for
>>> you?
>>>
>>>  -Sandy
>>>
>>>
>>> On Wed, Nov 13, 2013 at 4:19 PM, gaurav <[EMAIL PROTECTED]> wrote:
>>>
>>>> Hi Sandy,
>>>>
>>>> If I set relaxlocality to true with host name, I don't get the
>>>> container on specified host even though node has the resources.
>>>> I am using AMRMClient, only thing is I made following changes to get
>>>> the containers on the specified node.
>>>> /
>>>> //Line: 361 //
>>>> //Original//
>>>> // addResourceRequest(req.getPriority(), node, req.getCapability(),
>>>> req, true);//
>>>> //Modifiled//
>>>> // addResourceRequest(req.getPriority(), node, req.getCapability(),
>>>> req, req.getRelaxLocality());//
>>>> //
>>>> //Line 374//
>>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB