Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce >> mail # user >> Re: Allocating Containers on a particular Node in Yarn


Copy link to this message
-
Re: Allocating Containers on a particular Node in Yarn
What version are you using?  Setting the relax locality to true for nodes
is always correct.  For racks, this is not necessarily the case.  When I
look at trunk, it is set to true always on line 361 (which is correct), but
on on line 374.

-Sandy
On Wed, Nov 13, 2013 at 4:47 PM, gaurav <[EMAIL PROTECTED]> wrote:

>  Hi Sandy,
>
> No it is not working for me.  As mentioned earlier, AMRMClient is not
> respecting the locality for node and rack in *Line 361 and 374 *and is
> set to true always.
> I am requesting for one container with specified node and rack and relaxed
> locality false.
>
> Thanks
> -Gaurav
>
> On 11/13/2013 4:24 PM, Sandy Ryza wrote:
>
>  [moving to user list]
>
>  Right.  relaxLocality needs to be set on the next level up.  It
> determines whether locality can be relaxed to that level.  Confusing, I
> know.  If you are using AMRMClient, you should be able to accomplish what
> you're looking for by creating an AMRMClient.ContainerRequest that
> specifies a node and sets relax locality to false.  Is that not working for
> you?
>
>  -Sandy
>
>
> On Wed, Nov 13, 2013 at 4:19 PM, gaurav <[EMAIL PROTECTED]> wrote:
>
>> Hi Sandy,
>>
>> If I set relaxlocality to true with host name, I don't get the container
>> on specified host even though node has the resources.
>> I am using AMRMClient, only thing is I made following changes to get the
>> containers on the specified node.
>> /
>> //Line: 361 //
>> //Original//
>> // addResourceRequest(req.getPriority(), node, req.getCapability(), req,
>> true);//
>> //Modifiled//
>> // addResourceRequest(req.getPriority(), node, req.getCapability(), req,
>> req.getRelaxLocality());//
>> //
>> //Line 374//
>> //Original//
>> // addResourceRequest(req.getPriority(), rack, req.getCapability(), req,
>> true);//
>> //Modifiled//
>> // addResourceRequest(req.getPriority(), rack, req.getCapability(), req,
>> req.getRelaxLocality());//
>> ///
>>
>>
>> Thanks
>> -Gaurav
>>
>>
>> On 11/13/2013 4:02 PM, Sandy Ryza wrote:
>>
>>> Yeah, specifying a host name with relaxLocality is meaningful.
>>>  Schedulers
>>> use delay scheduling (
>>> http://www.cs.berkeley.edu/~matei/talks/2010/eurosys_delaysched.pdf) to
>>> achieve locality when relaxLocality is on.  But it is turned off by
>>> default.  The individual scheduler docs have specifics on how to
>>> configure
>>> it.
>>>
>>> Guarav,
>>> Using ResourceRequests directly is not straightforward and error prone.
>>>  Is
>>> there a reason that AMRMClient is unsuitable for your needs?
>>>
>>> -Sandy
>>>
>>>
>>>
>>> On Wed, Nov 13, 2013 at 3:55 PM, Thomas Weise <[EMAIL PROTECTED]
>>> >wrote:
>>>
>>>  Is it possible to specify a particular node and have RM fallback to an
>>>> different node only after making an attempt to allocate for the
>>>> requested
>>>> node? In other words, is the combination of specific host name and
>>>> relaxLocality=TRUE meaningful at all?
>>>>
>>>> Thanks.
>>>>
>>>>
>>>> On Wed, Nov 13, 2013 at 3:23 PM, Alejandro Abdelnur <[EMAIL PROTECTED]
>>>>
>>>>> wrote:
>>>>> Gaurav,
>>>>>
>>>>> Setting relaxLocality to FALSE should do it.
>>>>>
>>>>> thanks.
>>>>>
>>>>>
>>>>> On Wed, Nov 13, 2013 at 2:58 PM, gaurav <[EMAIL PROTECTED]>
>>>>> wrote:
>>>>>
>>>>>    Hi,
>>>>>>   I am trying to allocate containers on a particular node in Yarn but
>>>>>>
>>>>>  Yarn
>>>>
>>>>>  is returning me containers on different node although the requested
>>>>>>
>>>>>  node
>>>>
>>>>>  has resources available.
>>>>>>
>>>>>> I checked into the allocate(AllocateRequest request) function of
>>>>>> ApplicationMasterService and my request is as follows
>>>>>>
>>>>>> *request: ask { priority { priority: 1 } resource_name: "h2"
>>>>>>
>>>>>  capability {
>>>>
>>>>>  memory: 1000 } num_containers: 2 } ask { priority { priority: 1 }
>>>>>> resource_name: "/default-rack" capability { memory: 1000 }
>>>>>>
>>>>> num_containers:
>>>>>
>>>>>> 2 } ask { priority { priority: 1 } resource_name: "*" capability {
>>>>>>
>>>>> memory:
>>>>>
>>>>>> 1000 } num_containers: 2 } response_id: 1 progress: 0.0*