Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
MapReduce, mail # user - Re: Allocating Containers on a particular Node in Yarn


+
gaurav 2013-11-14, 00:47
+
Sandy Ryza 2013-11-14, 00:59
+
gaurav 2013-11-14, 01:04
+
gaurav 2013-11-14, 01:30
Copy link to this message
-
Re: Allocating Containers on a particular Node in Yarn
Sandy Ryza 2013-11-14, 03:36
In that case, the AMRMClient code looks correct to me.  Can you share the
code you've written against it that's not receiving the correct containers?
On Wed, Nov 13, 2013 at 5:30 PM, gaurav <[EMAIL PROTECTED]> wrote:

>  Hi,
>
> I looked in the trunk and line numbers are 361 and 366.
>
> Thanks
> -Gaurav
>
> On 11/13/2013 5:04 PM, gaurav wrote:
>
> I have hadoop-2.2.0
>
> Thanks
> -Gaurav
>
> On 11/13/2013 4:59 PM, Sandy Ryza wrote:
>
> What version are you using?  Setting the relax locality to true for nodes
> is always correct.  For racks, this is not necessarily the case.  When I
> look at trunk, it is set to true always on line 361 (which is correct), but
> on on line 374.
>
>  -Sandy
>
>
> On Wed, Nov 13, 2013 at 4:47 PM, gaurav <[EMAIL PROTECTED]> wrote:
>
>>  Hi Sandy,
>>
>> No it is not working for me.  As mentioned earlier, AMRMClient is not
>> respecting the locality for node and rack in *Line 361 and 374 *and is
>> set to true always.
>> I am requesting for one container with specified node and rack and
>> relaxed locality false.
>>
>> Thanks
>> -Gaurav
>>
>>  On 11/13/2013 4:24 PM, Sandy Ryza wrote:
>>
>>  [moving to user list]
>>
>>  Right.  relaxLocality needs to be set on the next level up.  It
>> determines whether locality can be relaxed to that level.  Confusing, I
>> know.  If you are using AMRMClient, you should be able to accomplish what
>> you're looking for by creating an AMRMClient.ContainerRequest that
>> specifies a node and sets relax locality to false.  Is that not working for
>> you?
>>
>>  -Sandy
>>
>>
>> On Wed, Nov 13, 2013 at 4:19 PM, gaurav <[EMAIL PROTECTED]> wrote:
>>
>>> Hi Sandy,
>>>
>>> If I set relaxlocality to true with host name, I don't get the container
>>> on specified host even though node has the resources.
>>> I am using AMRMClient, only thing is I made following changes to get the
>>> containers on the specified node.
>>> /
>>> //Line: 361 //
>>> //Original//
>>> // addResourceRequest(req.getPriority(), node, req.getCapability(), req,
>>> true);//
>>> //Modifiled//
>>> // addResourceRequest(req.getPriority(), node, req.getCapability(), req,
>>> req.getRelaxLocality());//
>>> //
>>> //Line 374//
>>> //Original//
>>> // addResourceRequest(req.getPriority(), rack, req.getCapability(), req,
>>> true);//
>>> //Modifiled//
>>> // addResourceRequest(req.getPriority(), rack, req.getCapability(), req,
>>> req.getRelaxLocality());//
>>> ///
>>>
>>>
>>> Thanks
>>> -Gaurav
>>>
>>>
>>> On 11/13/2013 4:02 PM, Sandy Ryza wrote:
>>>
>>>> Yeah, specifying a host name with relaxLocality is meaningful.
>>>>  Schedulers
>>>> use delay scheduling (
>>>> http://www.cs.berkeley.edu/~matei/talks/2010/eurosys_delaysched.pdf) to
>>>> achieve locality when relaxLocality is on.  But it is turned off by
>>>> default.  The individual scheduler docs have specifics on how to
>>>> configure
>>>> it.
>>>>
>>>> Guarav,
>>>> Using ResourceRequests directly is not straightforward and error prone.
>>>>  Is
>>>> there a reason that AMRMClient is unsuitable for your needs?
>>>>
>>>> -Sandy
>>>>
>>>>
>>>>
>>>> On Wed, Nov 13, 2013 at 3:55 PM, Thomas Weise <[EMAIL PROTECTED]
>>>> >wrote:
>>>>
>>>>  Is it possible to specify a particular node and have RM fallback to an
>>>>> different node only after making an attempt to allocate for the
>>>>> requested
>>>>> node? In other words, is the combination of specific host name and
>>>>> relaxLocality=TRUE meaningful at all?
>>>>>
>>>>> Thanks.
>>>>>
>>>>>
>>>>> On Wed, Nov 13, 2013 at 3:23 PM, Alejandro Abdelnur <[EMAIL PROTECTED]
>>>>>
>>>>>> wrote:
>>>>>> Gaurav,
>>>>>>
>>>>>> Setting relaxLocality to FALSE should do it.
>>>>>>
>>>>>> thanks.
>>>>>>
>>>>>>
>>>>>> On Wed, Nov 13, 2013 at 2:58 PM, gaurav <[EMAIL PROTECTED]>
>>>>>> wrote:
>>>>>>
>>>>>>    Hi,
>>>>>>>   I am trying to allocate containers on a particular node in Yarn but
>>>>>>>
>>>>>>  Yarn
>>>>>
>>>>>>  is returning me containers on different node although the requested
>>>>>>>
>>>>>>  node
+
gaurav 2013-11-14, 06:10
+
Sandy Ryza 2013-11-14, 19:41
+
Gaurav Gupta 2013-11-14, 20:05
+
Sandy Ryza 2013-11-14, 20:40
+
Gaurav Gupta 2013-11-14, 21:28
+
Sandy Ryza 2013-11-14, 21:33