Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce >> mail # user >> whitelist feature of YARN


Copy link to this message
-
Re: whitelist feature of YARN
Sandy,
  Thanks again. I found RC1 for 2.1.0-beta available at
http://people.apache.org/~acmurthy/hadoop-2.1.0-beta-rc1/
   Would this have the fix for YARN-521? and, can I use that?

-Kishore
On Wed, Aug 7, 2013 at 12:35 PM, Sandy Ryza <[EMAIL PROTECTED]> wrote:

> Responses inline:
>
>
> On Tue, Aug 6, 2013 at 11:55 PM, Krishna Kishore Bonagiri <
> [EMAIL PROTECTED]> wrote:
>
>> Hi Sandy,
>>
>>   Thanks for the reply and it is good to know YARN-521 is done! Please
>> answer my following questions
>>
>> 1) when is 2.1.0-beta going to be released? is it soon or do you suggest
>> me take it from the trunk or is there a recent release candidate available?
>>
>> We're very close and my guess would be no later than the end of the
> month (don't hold me to this).
>
>
>> 2) I have recently changed my application to use the new Asynchronous
>> interfaces. I am hoping it works with that too, correct me if I am wrong.
>>
> ContainerRequest is shared by the async interfaces as well so it should
> work here.
>
>>
>> 3) Change in interface:
>>
>> The old interface for ContainerRequest constructor used to be this:
>>
>>  public ContainerRequest(Resource capability, String[] nodes,
>>         String[] racks, Priority priority, int containerCount);
>>
>> where as now it is changed to
>>
>> a) public ContainerRequest(Resource capability, String[] nodes,
>>         String[] racks, Priority priority)
>> &
>>
>> b) public ContainerRequest(Resource capability, String[] nodes,
>>         String[] racks, Priority priority, boolean relaxLocality)
>>
>> that means the old argument containerCount is gone! How would I be able
>> to specify how many containers do I need?
>>
>> We now expect that you submit a ContainerRequest for each container you
> want.
>
>
>> -Kishore
>>
>>
>>
>>
>> On Wed, Aug 7, 2013 at 11:37 AM, Sandy Ryza <[EMAIL PROTECTED]>wrote:
>>
>>> YARN-521, which brings whitelisting to the AMRMClient APIs, is now
>>> included in 2.1.0-beta.  Check out the doc for the relaxLocality paramater
>>> in ContainerRequest in AMRMClient:
>>> https://github.com/apache/hadoop-common/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/AMRMClient.java and
>>> I can help clarify here if anything's confusing.
>>>
>>> -Sandy
>>>
>>>
>>> On Tue, Jul 9, 2013 at 2:54 AM, Krishna Kishore Bonagiri <
>>> [EMAIL PROTECTED]> wrote:
>>>
>>>> Hi Sandy,
>>>>
>>>>   Yes, I have been using AMRMClient APIs. I am planning to shift to
>>>> whatever way is this white list feature is supported with. But am not sure
>>>> what is meant by submitting ResourceRequests directly to RM. Can you please
>>>> elaborate on this or give me a pointer to some example code on how to do
>>>> it...
>>>>
>>>>    Thanks for the reply,
>>>>
>>>> -Kishore
>>>>
>>>>
>>>> On Mon, Jul 8, 2013 at 10:53 PM, Sandy Ryza <[EMAIL PROTECTED]>wrote:
>>>>
>>>>> Hi Krishna,
>>>>>
>>>>> From your previous email, it looks like you are using the AMRMClient
>>>>> APIs.  Support for whitelisting is not yet supported through them.  I am
>>>>> working on this in YARN-521, which should be included in the next release
>>>>> after 2.1.0-beta.  If you are submitting ResourceRequests directly to the
>>>>> RM, you can whitelist a node by
>>>>> * setting the relaxLocality flag on the node-level ResourceRequest to
>>>>> true
>>>>> * setting the relaxLocality flag on the corresponding rack-level
>>>>> ResourceRequest to false
>>>>> * setting the relaxLocality flag on the corresponding any-level
>>>>> ResourceRequest to false
>>>>>
>>>>> -Sandy
>>>>>
>>>>>
>>>>> On Mon, Jul 8, 2013 at 6:48 AM, Krishna Kishore Bonagiri <
>>>>> [EMAIL PROTECTED]> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>>   Can someone please point to some example code of how to use the
>>>>>> whitelist feature of YARN, I have recently got RC1 for hadoop-2.1.0-beta
>>>>>> and want to use this feature.
>>>>>>
>>>>>>   It would be great if you can point me to some description of what