Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
MapReduce, mail # user - whitelist feature of YARN


+
Krishna Kishore Bonagiri 2013-07-08, 13:48
+
Sandy Ryza 2013-07-08, 17:23
+
Krishna Kishore Bonagiri 2013-07-09, 09:54
+
Sandy Ryza 2013-08-07, 06:07
+
Krishna Kishore Bonagiri 2013-08-07, 06:55
Copy link to this message
-
Re: whitelist feature of YARN
Sandy Ryza 2013-08-07, 07:05
Responses inline:
On Tue, Aug 6, 2013 at 11:55 PM, Krishna Kishore Bonagiri <
[EMAIL PROTECTED]> wrote:

> Hi Sandy,
>
>   Thanks for the reply and it is good to know YARN-521 is done! Please
> answer my following questions
>
> 1) when is 2.1.0-beta going to be released? is it soon or do you suggest
> me take it from the trunk or is there a recent release candidate available?
>
> We're very close and my guess would be no later than the end of the
month (don't hold me to this).
> 2) I have recently changed my application to use the new Asynchronous
> interfaces. I am hoping it works with that too, correct me if I am wrong.
>
ContainerRequest is shared by the async interfaces as well so it should
work here.

>
> 3) Change in interface:
>
> The old interface for ContainerRequest constructor used to be this:
>
>  public ContainerRequest(Resource capability, String[] nodes,
>         String[] racks, Priority priority, int containerCount);
>
> where as now it is changed to
>
> a) public ContainerRequest(Resource capability, String[] nodes,
>         String[] racks, Priority priority)
> &
>
> b) public ContainerRequest(Resource capability, String[] nodes,
>         String[] racks, Priority priority, boolean relaxLocality)
>
> that means the old argument containerCount is gone! How would I be able to
> specify how many containers do I need?
>
> We now expect that you submit a ContainerRequest for each container you
want.
> -Kishore
>
>
>
>
> On Wed, Aug 7, 2013 at 11:37 AM, Sandy Ryza <[EMAIL PROTECTED]>wrote:
>
>> YARN-521, which brings whitelisting to the AMRMClient APIs, is now
>> included in 2.1.0-beta.  Check out the doc for the relaxLocality paramater
>> in ContainerRequest in AMRMClient:
>> https://github.com/apache/hadoop-common/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/AMRMClient.java and
>> I can help clarify here if anything's confusing.
>>
>> -Sandy
>>
>>
>> On Tue, Jul 9, 2013 at 2:54 AM, Krishna Kishore Bonagiri <
>> [EMAIL PROTECTED]> wrote:
>>
>>> Hi Sandy,
>>>
>>>   Yes, I have been using AMRMClient APIs. I am planning to shift to
>>> whatever way is this white list feature is supported with. But am not sure
>>> what is meant by submitting ResourceRequests directly to RM. Can you please
>>> elaborate on this or give me a pointer to some example code on how to do
>>> it...
>>>
>>>    Thanks for the reply,
>>>
>>> -Kishore
>>>
>>>
>>> On Mon, Jul 8, 2013 at 10:53 PM, Sandy Ryza <[EMAIL PROTECTED]>wrote:
>>>
>>>> Hi Krishna,
>>>>
>>>> From your previous email, it looks like you are using the AMRMClient
>>>> APIs.  Support for whitelisting is not yet supported through them.  I am
>>>> working on this in YARN-521, which should be included in the next release
>>>> after 2.1.0-beta.  If you are submitting ResourceRequests directly to the
>>>> RM, you can whitelist a node by
>>>> * setting the relaxLocality flag on the node-level ResourceRequest to
>>>> true
>>>> * setting the relaxLocality flag on the corresponding rack-level
>>>> ResourceRequest to false
>>>> * setting the relaxLocality flag on the corresponding any-level
>>>> ResourceRequest to false
>>>>
>>>> -Sandy
>>>>
>>>>
>>>> On Mon, Jul 8, 2013 at 6:48 AM, Krishna Kishore Bonagiri <
>>>> [EMAIL PROTECTED]> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>>   Can someone please point to some example code of how to use the
>>>>> whitelist feature of YARN, I have recently got RC1 for hadoop-2.1.0-beta
>>>>> and want to use this feature.
>>>>>
>>>>>   It would be great if you can point me to some description of what
>>>>> this white listing feature is, I have gone through some JIRA logs related
>>>>> to this but more concrete explanation would be helpful.
>>>>>
>>>>> Thanks,
>>>>> Kishore
>>>>>
>>>>
>>>>
>>>
>>
>
+
Krishna Kishore Bonagiri 2013-08-07, 09:06
+
Arun C Murthy 2013-08-07, 16:53