Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce, mail # user - Re: YARN - specify hosts in ContainerRequest


Copy link to this message
-
Re: YARN - specify hosts in ContainerRequest
Sandy Ryza 2013-04-12, 19:15
YARN currently does not provide any guarantees that a container will be
placed where you ask for it.  This is being addressed in YARN-392 and
YARN-398.  For now, the Fair Scheduler and Capacity Scheduler will make a
best effort to do this through delay scheduling.  If you use the Fair
Scheduler, you can turn on delay scheduling by setting
yarn.scheduler.fair.locality.threshold.node to a float between 0 and 1,
which corresponds to the number of node-RM heartbeats to delay allocating a
non-local container, as a fraction of the cluster size.

Sandy

On Fri, Apr 12, 2013 at 12:08 PM, Brian C. Huffman <
[EMAIL PROTECTED]> wrote:

>  I get a container, but not on the node I'm asking for.
>
> Thanks,
> Brian
>
>
> On 04/12/2013 03:01 PM, Sandy Ryza wrote:
>
> What do you mean when you say it doesn't seem to use the code?  That
> you're not getting containers back?
>
>  -Sandy
>
> On Fri, Apr 12, 2013 at 7:44 AM, Brian C. Huffman <
> [EMAIL PROTECTED]> wrote:
>
>> Yes, this is exactly the DistributedShell example in the 2.0.3 source
>> (except for the added specification of containerNode).
>>
>> So, it's doing the following:
>>           ContainerRequest containerAsk >> setupContainerAskForRM(askCount);
>>           resourceManager.addContainerRequest(containerAsk);
>>
>>
>>         // Send the request to RM
>>         LOG.info("Asking RM for containers" + ", askCount=" + askCount);
>>         AMResponse amResp = sendContainerAskToRM();
>>
>> FYI - the sendContainerAskToRM calls the allocate(progress).
>>
>> I don't see any NPEs in the resourcemanager's logs.
>>
>> Regards,
>> Brian
>>
>>
>> On 04/11/2013 02:50 PM, Harsh J wrote:
>>
>>> Hi Brian,
>>>
>>> Your way of creation seems fine. Are you "adding" this to the
>>> AMRMClient somewhere though, via [1], right before you call
>>> allocate(progress)? Cause otherwise this object's not used at all.
>>>
>>> P.s. I'm not sure we'll handle that null for racks all that well in
>>> certain places yet; I've noticed errors in my app when I've missed
>>> passing elements or passed them as nulls, but I didn't rebase it onto
>>> 2.0.3 yet so haven't reported anything so far - will do so soon. I'd
>>> check the RM log to see if there's any WARN logged there with an NPE.
>>>
>>> [1] -
>>> http://hadoop.apache.org/docs/current/api/org/apache/hadoop/yarn/client/AMRMClient.html#addContainerRequest(org.apache.hadoop.yarn.client.AMRMClient.ContainerRequest)
>>>
>>> On Thu, Apr 11, 2013 at 11:30 PM, Brian C. Huffman
>>> <[EMAIL PROTECTED]> wrote:
>>>
>>>> All,
>>>>
>>>> I'm starting to work with YARN and I'm trying to make some changes to
>>>> the
>>>> distributedshell example.
>>>>
>>>> Specifically I'm trying to set the host where the distributed shell
>>>> container will run.  I added the following code into
>>>> ApplicationMaster.java:
>>>>      final String[] containerNodes = { "node1" };
>>>>
>>>>      LOG.info("Trying to request node: " + containerNodes[0]);
>>>>      ContainerRequest request = new ContainerRequest(capability,
>>>> containerNodes, null,
>>>>          pri, numContainers);
>>>>
>>>> And while the log output shows this code is being executed ("Trying to
>>>> request..."), it doesn't seem to use it.  Is this functionality that
>>>> isn't
>>>> complete yet in the 2.0.3 yarn alpha?  Or am I doing something wrong?
>>>>
>>>> Thanks,
>>>> Brian
>>>>
>>>>
>>>
>>>
>>
>>
>
>