Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce, mail # user - whitelist feature of YARN


Copy link to this message
-
Re: whitelist feature of YARN
Arun C Murthy 2013-08-07, 16:53
Go ahead - it's very close, I'll probably create an rc2 in the next couple of days.

On Aug 7, 2013, at 2:06 AM, Krishna Kishore Bonagiri <[EMAIL PROTECTED]> wrote:

> Sandy,
>   Thanks again. I found RC1 for 2.1.0-beta available at http://people.apache.org/~acmurthy/hadoop-2.1.0-beta-rc1/
>    Would this have the fix for YARN-521? and, can I use that?
>
> -Kishore
>
>
> On Wed, Aug 7, 2013 at 12:35 PM, Sandy Ryza <[EMAIL PROTECTED]> wrote:
> Responses inline:
>
>
> On Tue, Aug 6, 2013 at 11:55 PM, Krishna Kishore Bonagiri <[EMAIL PROTECTED]> wrote:
> Hi Sandy,
>
>   Thanks for the reply and it is good to know YARN-521 is done! Please answer my following questions
>
> 1) when is 2.1.0-beta going to be released? is it soon or do you suggest me take it from the trunk or is there a recent release candidate available?
>
> We're very close and my guess would be no later than the end of the month (don't hold me to this).
>  
> 2) I have recently changed my application to use the new Asynchronous interfaces. I am hoping it works with that too, correct me if I am wrong.
> ContainerRequest is shared by the async interfaces as well so it should work here.
>
> 3) Change in interface:
>
> The old interface for ContainerRequest constructor used to be this:
>
>  public ContainerRequest(Resource capability, String[] nodes,
>         String[] racks, Priority priority, int containerCount);
>
> where as now it is changed to
>
> a) public ContainerRequest(Resource capability, String[] nodes,
>         String[] racks, Priority priority)
> &
>
> b) public ContainerRequest(Resource capability, String[] nodes,
>         String[] racks, Priority priority, boolean relaxLocality)
>
> that means the old argument containerCount is gone! How would I be able to specify how many containers do I need?
>
> We now expect that you submit a ContainerRequest for each container you want.
>  
> -Kishore
>
>
>
>
> On Wed, Aug 7, 2013 at 11:37 AM, Sandy Ryza <[EMAIL PROTECTED]> wrote:
> YARN-521, which brings whitelisting to the AMRMClient APIs, is now included in 2.1.0-beta.  Check out the doc for the relaxLocality paramater in ContainerRequest in AMRMClient: https://github.com/apache/hadoop-common/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/AMRMClient.java and I can help clarify here if anything's confusing.
>
> -Sandy
>
>
> On Tue, Jul 9, 2013 at 2:54 AM, Krishna Kishore Bonagiri <[EMAIL PROTECTED]> wrote:
> Hi Sandy,
>
>   Yes, I have been using AMRMClient APIs. I am planning to shift to whatever way is this white list feature is supported with. But am not sure what is meant by submitting ResourceRequests directly to RM. Can you please elaborate on this or give me a pointer to some example code on how to do it...
>
>    Thanks for the reply,
>
> -Kishore
>
>
> On Mon, Jul 8, 2013 at 10:53 PM, Sandy Ryza <[EMAIL PROTECTED]> wrote:
> Hi Krishna,
>
> From your previous email, it looks like you are using the AMRMClient APIs.  Support for whitelisting is not yet supported through them.  I am working on this in YARN-521, which should be included in the next release after 2.1.0-beta.  If you are submitting ResourceRequests directly to the RM, you can whitelist a node by
> * setting the relaxLocality flag on the node-level ResourceRequest to true
> * setting the relaxLocality flag on the corresponding rack-level ResourceRequest to false
> * setting the relaxLocality flag on the corresponding any-level ResourceRequest to false
>
> -Sandy
>
>
> On Mon, Jul 8, 2013 at 6:48 AM, Krishna Kishore Bonagiri <[EMAIL PROTECTED]> wrote:
> Hi,
>
>   Can someone please point to some example code of how to use the whitelist feature of YARN, I have recently got RC1 for hadoop-2.1.0-beta and want to use this feature.
>
>   It would be great if you can point me to some description of what this white listing feature is, I have gone through some JIRA logs related to this but more concrete explanation would be helpful.

Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/