Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce >> mail # user >> Re: Issues with ResourceManager scheduling functions


Copy link to this message
-
Re: Issues with ResourceManager scheduling functions
Try requesting all your 100GB containers with priority 1 and your 50 GB
containers with priority 2.

-Sandy
On Mon, Dec 9, 2013 at 12:59 AM, Bill Q <[EMAIL PROTECTED]> wrote:

> Thanks a lot.
>
> So, in order for the two types of containers mentioned above to work, what
> priorities should I assign to them? Can you give an example?
>
> Many thanks.
>
>
> Bill
>
>
> On Mon, Dec 9, 2013 at 3:30 AM, Sandy Ryza <[EMAIL PROTECTED]>wrote:
>
>> YARN currently is unable to handle requests with different resource
>> requirements at the same priority (YARN-314).  Using different priorities
>> would likely solve the issue.  I believe it would also be safest to avoid
>> using the priority 0 because this is used for the application master.
>>
>> -Sandy
>>
>>
>> On Mon, Dec 9, 2013 at 12:23 AM, Bill Q <[EMAIL PROTECTED]> wrote:
>>
>>> No, all default 0. Should I use different priorities?
>>>
>>> Many thanks.
>>>
>>>
>>> Bill
>>>
>>>
>>> On Mon, Dec 9, 2013 at 2:53 AM, Sandy Ryza <[EMAIL PROTECTED]>wrote:
>>>
>>>> Are you requesting the different container types at different
>>>> priorities?
>>>>
>>>> -Sandy
>>>>
>>>>
>>>> On Sun, Dec 8, 2013 at 8:52 PM, Bill Q <[EMAIL PROTECTED]> wrote:
>>>>
>>>>> Hi,
>>>>> I recently implemented an application using YARN. Everything works
>>>>> fine, except for the illogical scheduling behavior of the ResourceManger,
>>>>> which might be caused by the heterogeneity of the cluster.
>>>>>
>>>>> I have two types of computing node in the cluster. 10 nodes with 128G
>>>>> memory and 10 nodes with 64G memory. In the application, I create two types
>>>>> of container. Type A container would ask for 50G of memory and Type B
>>>>> container would ask for 100G of memory.
>>>>>
>>>>> The problem is that if I created 50 Type B containers (100G), and 50
>>>>> Type A containers (50G), Only Type B container (100G) are running. Instead
>>>>> of running the Type A container (50G), all the nodes with 64G memory are
>>>>> idling. And only the nodes with 128G memory are running Type B container
>>>>> (100G). This doesn't make too much sense to me.
>>>>>
>>>>> If I only created Type A containers (50G), all the nodes would be able
>>>>> run the containers. The 128G node would run two containers simultaneously.
>>>>> So, the problem seems to be connected to the different types of containers.
>>>>>
>>>>> Does anybody have any suggestions to fix this issue?
>>>>>
>>>>> Many thanks.
>>>>>
>>>>>
>>>>> Bill
>>>>>
>>>>
>>>>
>>>
>>
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB