Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HDFS >> mail # user >> Allocating Containers on a particular Node in Yarn


Copy link to this message
-
Re: Allocating Containers on a particular Node in Yarn
[moving to user list]

Right.  relaxLocality needs to be set on the next level up.  It determines
whether locality can be relaxed to that level.  Confusing, I know.  If you
are using AMRMClient, you should be able to accomplish what you're looking
for by creating an AMRMClient.ContainerRequest that specifies a node and
sets relax locality to false.  Is that not working for you?

-Sandy
On Wed, Nov 13, 2013 at 4:19 PM, gaurav <[EMAIL PROTECTED]> wrote:

> Hi Sandy,
>
> If I set relaxlocality to true with host name, I don't get the container
> on specified host even though node has the resources.
> I am using AMRMClient, only thing is I made following changes to get the
> containers on the specified node.
> /
> //Line: 361 //
> //Original//
> // addResourceRequest(req.getPriority(), node, req.getCapability(), req,
> true);//
> //Modifiled//
> // addResourceRequest(req.getPriority(), node, req.getCapability(), req,
> req.getRelaxLocality());//
> //
> //Line 374//
> //Original//
> // addResourceRequest(req.getPriority(), rack, req.getCapability(), req,
> true);//
> //Modifiled//
> // addResourceRequest(req.getPriority(), rack, req.getCapability(), req,
> req.getRelaxLocality());//
> ///
>
>
> Thanks
> -Gaurav
>
>
> On 11/13/2013 4:02 PM, Sandy Ryza wrote:
>
>> Yeah, specifying a host name with relaxLocality is meaningful.  Schedulers
>> use delay scheduling (
>> http://www.cs.berkeley.edu/~matei/talks/2010/eurosys_delaysched.pdf) to
>> achieve locality when relaxLocality is on.  But it is turned off by
>> default.  The individual scheduler docs have specifics on how to configure
>> it.
>>
>> Guarav,
>> Using ResourceRequests directly is not straightforward and error prone.
>>  Is
>> there a reason that AMRMClient is unsuitable for your needs?
>>
>> -Sandy
>>
>>
>>
>> On Wed, Nov 13, 2013 at 3:55 PM, Thomas Weise <[EMAIL PROTECTED]
>> >wrote:
>>
>>  Is it possible to specify a particular node and have RM fallback to an
>>> different node only after making an attempt to allocate for the requested
>>> node? In other words, is the combination of specific host name and
>>> relaxLocality=TRUE meaningful at all?
>>>
>>> Thanks.
>>>
>>>
>>> On Wed, Nov 13, 2013 at 3:23 PM, Alejandro Abdelnur <[EMAIL PROTECTED]
>>>
>>>> wrote:
>>>> Gaurav,
>>>>
>>>> Setting relaxLocality to FALSE should do it.
>>>>
>>>> thanks.
>>>>
>>>>
>>>> On Wed, Nov 13, 2013 at 2:58 PM, gaurav <[EMAIL PROTECTED]> wrote:
>>>>
>>>>    Hi,
>>>>>   I am trying to allocate containers on a particular node in Yarn but
>>>>>
>>>> Yarn
>>>
>>>> is returning me containers on different node although the requested
>>>>>
>>>> node
>>>
>>>> has resources available.
>>>>>
>>>>> I checked into the allocate(AllocateRequest request) function of
>>>>> ApplicationMasterService and my request is as follows
>>>>>
>>>>> *request: ask { priority { priority: 1 } resource_name: "h2"
>>>>>
>>>> capability {
>>>
>>>> memory: 1000 } num_containers: 2 } ask { priority { priority: 1 }
>>>>> resource_name: "/default-rack" capability { memory: 1000 }
>>>>>
>>>> num_containers:
>>>>
>>>>> 2 } ask { priority { priority: 1 } resource_name: "*" capability {
>>>>>
>>>> memory:
>>>>
>>>>> 1000 } num_containers: 2 } response_id: 1 progress: 0.0*
>>>>>
>>>>> but the containers that I am getting back is as follows
>>>>> [Container: [ContainerId: container_1384381084244_0001_01_000002,
>>>>>
>>>> NodeId:
>>>
>>>> h1:1234, NodeHttpAddress: h1:2, Resource: <memory:1024, vCores:1>,
>>>>> Priority: 1, Token: Token { kind: ContainerToken, service: h1:1234 },
>>>>>
>>>> ],
>>>
>>>> Container: [ContainerId: container_1384381084244_0001_01_000003,
>>>>>
>>>> NodeId:
>>>
>>>> h1:1234, NodeHttpAddress: h1:2, Resource: <memory:1024, vCores:1>,
>>>>> Priority: 1, Token: Token { kind: ContainerToken, service: h1:1234 },
>>>>>
>>>> ]]
>>>
>>>> I am attaching the test case that I have written along with the mail.
>>>>>
>>>> It
>>>
>>>> uses classes under org.apache.hadoop.yarn.server.resourcemanager
>>>>>
>>