Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce, mail # user - Re: Issues with ResourceManager scheduling functions


Copy link to this message
-
Re: Issues with ResourceManager scheduling functions
Sandy Ryza 2013-12-09, 09:02
Try requesting all your 100GB containers with priority 1 and your 50 GB
containers with priority 2.

-Sandy
On Mon, Dec 9, 2013 at 12:59 AM, Bill Q <[EMAIL PROTECTED]> wrote:

> Thanks a lot.
>
> So, in order for the two types of containers mentioned above to work, what
> priorities should I assign to them? Can you give an example?
>
> Many thanks.
>
>
> Bill
>
>
> On Mon, Dec 9, 2013 at 3:30 AM, Sandy Ryza <[EMAIL PROTECTED]>wrote:
>
>> YARN currently is unable to handle requests with different resource
>> requirements at the same priority (YARN-314).  Using different priorities
>> would likely solve the issue.  I believe it would also be safest to avoid
>> using the priority 0 because this is used for the application master.
>>
>> -Sandy
>>
>>
>> On Mon, Dec 9, 2013 at 12:23 AM, Bill Q <[EMAIL PROTECTED]> wrote:
>>
>>> No, all default 0. Should I use different priorities?
>>>
>>> Many thanks.
>>>
>>>
>>> Bill
>>>
>>>
>>> On Mon, Dec 9, 2013 at 2:53 AM, Sandy Ryza <[EMAIL PROTECTED]>wrote:
>>>
>>>> Are you requesting the different container types at different
>>>> priorities?
>>>>
>>>> -Sandy
>>>>
>>>>
>>>> On Sun, Dec 8, 2013 at 8:52 PM, Bill Q <[EMAIL PROTECTED]> wrote:
>>>>
>>>>> Hi,
>>>>> I recently implemented an application using YARN. Everything works
>>>>> fine, except for the illogical scheduling behavior of the ResourceManger,
>>>>> which might be caused by the heterogeneity of the cluster.
>>>>>
>>>>> I have two types of computing node in the cluster. 10 nodes with 128G
>>>>> memory and 10 nodes with 64G memory. In the application, I create two types
>>>>> of container. Type A container would ask for 50G of memory and Type B
>>>>> container would ask for 100G of memory.
>>>>>
>>>>> The problem is that if I created 50 Type B containers (100G), and 50
>>>>> Type A containers (50G), Only Type B container (100G) are running. Instead
>>>>> of running the Type A container (50G), all the nodes with 64G memory are
>>>>> idling. And only the nodes with 128G memory are running Type B container
>>>>> (100G). This doesn't make too much sense to me.
>>>>>
>>>>> If I only created Type A containers (50G), all the nodes would be able
>>>>> run the containers. The 128G node would run two containers simultaneously.
>>>>> So, the problem seems to be connected to the different types of containers.
>>>>>
>>>>> Does anybody have any suggestions to fix this issue?
>>>>>
>>>>> Many thanks.
>>>>>
>>>>>
>>>>> Bill
>>>>>
>>>>
>>>>
>>>
>>
>