Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HDFS >> mail # user >> Allocating Containers on a particular Node in Yarn


Copy link to this message
-
Re: Allocating Containers on a particular Node in Yarn
[moving to user list]

Right.  relaxLocality needs to be set on the next level up.  It determines
whether locality can be relaxed to that level.  Confusing, I know.  If you
are using AMRMClient, you should be able to accomplish what you're looking
for by creating an AMRMClient.ContainerRequest that specifies a node and
sets relax locality to false.  Is that not working for you?

-Sandy
On Wed, Nov 13, 2013 at 4:19 PM, gaurav <[EMAIL PROTECTED]> wrote:

> Hi Sandy,
>
> If I set relaxlocality to true with host name, I don't get the container
> on specified host even though node has the resources.
> I am using AMRMClient, only thing is I made following changes to get the
> containers on the specified node.
> /
> //Line: 361 //
> //Original//
> // addResourceRequest(req.getPriority(), node, req.getCapability(), req,
> true);//
> //Modifiled//
> // addResourceRequest(req.getPriority(), node, req.getCapability(), req,
> req.getRelaxLocality());//
> //
> //Line 374//
> //Original//
> // addResourceRequest(req.getPriority(), rack, req.getCapability(), req,
> true);//
> //Modifiled//
> // addResourceRequest(req.getPriority(), rack, req.getCapability(), req,
> req.getRelaxLocality());//
> ///
>
>
> Thanks
> -Gaurav
>
>
> On 11/13/2013 4:02 PM, Sandy Ryza wrote:
>
>> Yeah, specifying a host name with relaxLocality is meaningful.  Schedulers
>> use delay scheduling (
>> http://www.cs.berkeley.edu/~matei/talks/2010/eurosys_delaysched.pdf) to
>> achieve locality when relaxLocality is on.  But it is turned off by
>> default.  The individual scheduler docs have specifics on how to configure
>> it.
>>
>> Guarav,
>> Using ResourceRequests directly is not straightforward and error prone.
>>  Is
>> there a reason that AMRMClient is unsuitable for your needs?
>>
>> -Sandy
>>
>>
>>
>> On Wed, Nov 13, 2013 at 3:55 PM, Thomas Weise <[EMAIL PROTECTED]
>> >wrote:
>>
>>  Is it possible to specify a particular node and have RM fallback to an
>>> different node only after making an attempt to allocate for the requested
>>> node? In other words, is the combination of specific host name and
>>> relaxLocality=TRUE meaningful at all?
>>>
>>> Thanks.
>>>
>>>
>>> On Wed, Nov 13, 2013 at 3:23 PM, Alejandro Abdelnur <[EMAIL PROTECTED]
>>>
>>>> wrote:
>>>> Gaurav,
>>>>
>>>> Setting relaxLocality to FALSE should do it.
>>>>
>>>> thanks.
>>>>
>>>>
>>>> On Wed, Nov 13, 2013 at 2:58 PM, gaurav <[EMAIL PROTECTED]> wrote:
>>>>
>>>>    Hi,
>>>>>   I am trying to allocate containers on a particular node in Yarn but
>>>>>
>>>> Yarn
>>>
>>>> is returning me containers on different node although the requested
>>>>>
>>>> node
>>>
>>>> has resources available.
>>>>>
>>>>> I checked into the allocate(AllocateRequest request) function of
>>>>> ApplicationMasterService and my request is as follows
>>>>>
>>>>> *request: ask { priority { priority: 1 } resource_name: "h2"
>>>>>
>>>> capability {
>>>
>>>> memory: 1000 } num_containers: 2 } ask { priority { priority: 1 }
>>>>> resource_name: "/default-rack" capability { memory: 1000 }
>>>>>
>>>> num_containers:
>>>>
>>>>> 2 } ask { priority { priority: 1 } resource_name: "*" capability {
>>>>>
>>>> memory:
>>>>
>>>>> 1000 } num_containers: 2 } response_id: 1 progress: 0.0*
>>>>>
>>>>> but the containers that I am getting back is as follows
>>>>> [Container: [ContainerId: container_1384381084244_0001_01_000002,
>>>>>
>>>> NodeId:
>>>
>>>> h1:1234, NodeHttpAddress: h1:2, Resource: <memory:1024, vCores:1>,
>>>>> Priority: 1, Token: Token { kind: ContainerToken, service: h1:1234 },
>>>>>
>>>> ],
>>>
>>>> Container: [ContainerId: container_1384381084244_0001_01_000003,
>>>>>
>>>> NodeId:
>>>
>>>> h1:1234, NodeHttpAddress: h1:2, Resource: <memory:1024, vCores:1>,
>>>>> Priority: 1, Token: Token { kind: ContainerToken, service: h1:1234 },
>>>>>
>>>> ]]
>>>
>>>> I am attaching the test case that I have written along with the mail.
>>>>>
>>>> It
>>>
>>>> uses classes under org.apache.hadoop.yarn.server.resourcemanager
>>>>>
>>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB