Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce >> mail # user >> whitelist feature of YARN


Copy link to this message
-
Re: whitelist feature of YARN
Sandy,
  Thanks again. I found RC1 for 2.1.0-beta available at
http://people.apache.org/~acmurthy/hadoop-2.1.0-beta-rc1/
   Would this have the fix for YARN-521? and, can I use that?

-Kishore
On Wed, Aug 7, 2013 at 12:35 PM, Sandy Ryza <[EMAIL PROTECTED]> wrote:

> Responses inline:
>
>
> On Tue, Aug 6, 2013 at 11:55 PM, Krishna Kishore Bonagiri <
> [EMAIL PROTECTED]> wrote:
>
>> Hi Sandy,
>>
>>   Thanks for the reply and it is good to know YARN-521 is done! Please
>> answer my following questions
>>
>> 1) when is 2.1.0-beta going to be released? is it soon or do you suggest
>> me take it from the trunk or is there a recent release candidate available?
>>
>> We're very close and my guess would be no later than the end of the
> month (don't hold me to this).
>
>
>> 2) I have recently changed my application to use the new Asynchronous
>> interfaces. I am hoping it works with that too, correct me if I am wrong.
>>
> ContainerRequest is shared by the async interfaces as well so it should
> work here.
>
>>
>> 3) Change in interface:
>>
>> The old interface for ContainerRequest constructor used to be this:
>>
>>  public ContainerRequest(Resource capability, String[] nodes,
>>         String[] racks, Priority priority, int containerCount);
>>
>> where as now it is changed to
>>
>> a) public ContainerRequest(Resource capability, String[] nodes,
>>         String[] racks, Priority priority)
>> &
>>
>> b) public ContainerRequest(Resource capability, String[] nodes,
>>         String[] racks, Priority priority, boolean relaxLocality)
>>
>> that means the old argument containerCount is gone! How would I be able
>> to specify how many containers do I need?
>>
>> We now expect that you submit a ContainerRequest for each container you
> want.
>
>
>> -Kishore
>>
>>
>>
>>
>> On Wed, Aug 7, 2013 at 11:37 AM, Sandy Ryza <[EMAIL PROTECTED]>wrote:
>>
>>> YARN-521, which brings whitelisting to the AMRMClient APIs, is now
>>> included in 2.1.0-beta.  Check out the doc for the relaxLocality paramater
>>> in ContainerRequest in AMRMClient:
>>> https://github.com/apache/hadoop-common/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/AMRMClient.java and
>>> I can help clarify here if anything's confusing.
>>>
>>> -Sandy
>>>
>>>
>>> On Tue, Jul 9, 2013 at 2:54 AM, Krishna Kishore Bonagiri <
>>> [EMAIL PROTECTED]> wrote:
>>>
>>>> Hi Sandy,
>>>>
>>>>   Yes, I have been using AMRMClient APIs. I am planning to shift to
>>>> whatever way is this white list feature is supported with. But am not sure
>>>> what is meant by submitting ResourceRequests directly to RM. Can you please
>>>> elaborate on this or give me a pointer to some example code on how to do
>>>> it...
>>>>
>>>>    Thanks for the reply,
>>>>
>>>> -Kishore
>>>>
>>>>
>>>> On Mon, Jul 8, 2013 at 10:53 PM, Sandy Ryza <[EMAIL PROTECTED]>wrote:
>>>>
>>>>> Hi Krishna,
>>>>>
>>>>> From your previous email, it looks like you are using the AMRMClient
>>>>> APIs.  Support for whitelisting is not yet supported through them.  I am
>>>>> working on this in YARN-521, which should be included in the next release
>>>>> after 2.1.0-beta.  If you are submitting ResourceRequests directly to the
>>>>> RM, you can whitelist a node by
>>>>> * setting the relaxLocality flag on the node-level ResourceRequest to
>>>>> true
>>>>> * setting the relaxLocality flag on the corresponding rack-level
>>>>> ResourceRequest to false
>>>>> * setting the relaxLocality flag on the corresponding any-level
>>>>> ResourceRequest to false
>>>>>
>>>>> -Sandy
>>>>>
>>>>>
>>>>> On Mon, Jul 8, 2013 at 6:48 AM, Krishna Kishore Bonagiri <
>>>>> [EMAIL PROTECTED]> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>>   Can someone please point to some example code of how to use the
>>>>>> whitelist feature of YARN, I have recently got RC1 for hadoop-2.1.0-beta
>>>>>> and want to use this feature.
>>>>>>
>>>>>>   It would be great if you can point me to some description of what
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB