Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce, mail # user - Re: Allocating Containers on a particular Node in Yarn


Copy link to this message
-
Re: Allocating Containers on a particular Node in Yarn
Sandy Ryza 2013-11-14, 19:41
Requesting the rack is not necessary, and is leading to the behavior that
you're seeing.

The documentation states:
   * <li>If locality relaxation is disabled, then only within the same
request,
   * a node and its rack may be specified together. This allows for a
specific
   * rack with a preference for a specific node within that rack.</li>

On Wed, Nov 13, 2013 at 10:10 PM, gaurav <[EMAIL PROTECTED]> wrote:

>  Here is the snippet of code that I am using to allocate containers
>
>     AMRMClient<ContainerRequest> amRmClient > AMRMClient.createAMRMClient();;
>     String host = "h1";
>     Resource capability = Records.newRecord(Resource.class);
>     capability.setMemory(memory);
>     nodes = new String[] {host};
>     // in order to request a host, we also have to request the rack
>     racks = new String[] {"/default-rack"};
>      List<ContainerRequest> containerRequests = new
> ArrayList<ContainerRequest>();
>     List<ContainerId> releasedContainers = new ArrayList<ContainerId>();
>     containerRequests.add(new ContainerRequest(capability, nodes, racks,
> Priority.newInstance(priority),false));
>     if (containerRequests.size() > 0) {
>       LOG.info("Asking RM for containers: " + containerRequests);
>       for (ContainerRequest cr : containerRequests) {
>         LOG.info("Requested container: {}", cr.toString());
>         amRmClient.addContainerRequest(cr);
>       }
>     }
>
>     for (ContainerId containerId : releasedContainers) {
>       LOG.info("Released container, id={}", containerId.getId());
>       amRmClient.releaseAssignedContainer(containerId);
>     }
>     return amRmClient.allocate(0);
>
>
>
> -Gaurav
>
>
> On 11/13/2013 07:36 PM, Sandy Ryza wrote:
>
> In that case, the AMRMClient code looks correct to me.  Can you share the
> code you've written against it that's not receiving the correct containers?
>
>
>  On Wed, Nov 13, 2013 at 5:30 PM, gaurav <[EMAIL PROTECTED]> wrote:
>
>>  Hi,
>>
>> I looked in the trunk and line numbers are 361 and 366.
>>
>> Thanks
>> -Gaurav
>>
>>  On 11/13/2013 5:04 PM, gaurav wrote:
>>
>> I have hadoop-2.2.0
>>
>> Thanks
>> -Gaurav
>>
>> On 11/13/2013 4:59 PM, Sandy Ryza wrote:
>>
>> What version are you using?  Setting the relax locality to true for nodes
>> is always correct.  For racks, this is not necessarily the case.  When I
>> look at trunk, it is set to true always on line 361 (which is correct), but
>> on on line 374.
>>
>>  -Sandy
>>
>>
>> On Wed, Nov 13, 2013 at 4:47 PM, gaurav <[EMAIL PROTECTED]> wrote:
>>
>>>  Hi Sandy,
>>>
>>> No it is not working for me.  As mentioned earlier, AMRMClient is not
>>> respecting the locality for node and rack in *Line 361 and 374 *and is
>>> set to true always.
>>> I am requesting for one container with specified node and rack and
>>> relaxed locality false.
>>>
>>> Thanks
>>> -Gaurav
>>>
>>>  On 11/13/2013 4:24 PM, Sandy Ryza wrote:
>>>
>>>  [moving to user list]
>>>
>>>  Right.  relaxLocality needs to be set on the next level up.  It
>>> determines whether locality can be relaxed to that level.  Confusing, I
>>> know.  If you are using AMRMClient, you should be able to accomplish what
>>> you're looking for by creating an AMRMClient.ContainerRequest that
>>> specifies a node and sets relax locality to false.  Is that not working for
>>> you?
>>>
>>>  -Sandy
>>>
>>>
>>> On Wed, Nov 13, 2013 at 4:19 PM, gaurav <[EMAIL PROTECTED]> wrote:
>>>
>>>> Hi Sandy,
>>>>
>>>> If I set relaxlocality to true with host name, I don't get the
>>>> container on specified host even though node has the resources.
>>>> I am using AMRMClient, only thing is I made following changes to get
>>>> the containers on the specified node.
>>>> /
>>>> //Line: 361 //
>>>> //Original//
>>>> // addResourceRequest(req.getPriority(), node, req.getCapability(),
>>>> req, true);//
>>>> //Modifiled//
>>>> // addResourceRequest(req.getPriority(), node, req.getCapability(),
>>>> req, req.getRelaxLocality());//
>>>> //
>>>> //Line 374//
>>