Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce >> mail # user >> Re: Parameter 'yarn.nodemanager.resource.cpu-cores' does not work


Copy link to this message
-
Re: Parameter 'yarn.nodemanager.resource.cpu-cores' does not work
Thanks, but seems it does not work for me.

My hadoop version is 'Hadoop 2.0.4-alpha', and seems it does not include a
class named
'org.apache.hadoop.yarn.util.resource.DominantResourceCalculator', so I
replaced it with
'org.apache.hadoop.yarn.util.LinuxResourceCalculatorPlugin'. My
configuration is below, and use the default values for other
LinuxContainerExecutor configurations. In my expectation, if I set the
value of 'yarn.nodemanager.resource.cpu-cores' to 0, no job could be
completed by nodemanager as the slaves do not have any cpu to use. But, in
fact, my job could be completed successfully. Why?

 <property>
    <name>yarn.nodemanager.resource.cpu-cores</name>
    <value>0</value>
  </property>

 <property>
    <name>yarn.nodemanager.vcores-pcores-ratio</name>
    <value>0</value>
  </property>

 <property>

<name>yarn.nodemanager.container-monitor.resource-calculator.class</name>
    <value>org.apache.hadoop.yarn.util.LinuxResourceCalculatorPlugin</value>
  </property>

 <property>
    <name>yarn.nodemanager.container-executor.class</name>

<value>org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor</value>
  </property>

2013/7/4 Chuan Liu <[EMAIL PROTECTED]>

>  I think you need to change the following configurations in yarn-site.xml
> to enable CPU resource limits.****
>
> ** **
>
> ‘yarn.nodemanager.container-monitor.resource-calculator.class’
> ‘org.apache.hadoop.yarn.util.resource.DominantResourceCalculator’****
>
> ** **
>
> ‘yarn.nodemanager.container-executor.class’****
>
> ‘org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor’****
>
> ** **
>
> Some LinuxContainerExecutor configurations:****
>
> yarn.nodemanager.linux-container-executor.path****
>
> yarn.nodemanager.linux-container-executor.resources-handler.class****
>
> yarn.nodemanager.linux-container-executor.cgroups.hierarchy****
>
> yarn.nodemanager.linux-container-executor.cgroups.mount****
>
> yarn.nodemanager.linux-container-executor.cgroups.mount-path****
>
> ** **
>
> -Chuan****
>
> ** **
>
> *From:* sam liu [mailto:[EMAIL PROTECTED]]
> *Sent:* Tuesday, July 02, 2013 8:33 PM
> *To:* [EMAIL PROTECTED]
> *Subject:* Parameter 'yarn.nodemanager.resource.cpu-cores' does not work**
> **
>
> ** **
>
> Hi,****
>
> With Hadoop 2.0.4-alpha, yarn.nodemanager.resource.cpu-cores does not work
> for me:****
>
> 1. The performance of running same terasort job do not change, even after
> increasing or decreasing the value of 'yarn.nodemanager.resource.cpu-cores'
> in yarn-site.xml and restart the yarn cluster.  ****
>
> 2. Even if I set the value of both
> 'yarn.nodemanager.resource.cpu-cores' and
> 'yarn.nodemanager.vcores-pcores-ratio' to 0, the MR job still could
> complete without any exception, but the expected behavior should be that no
> cpu could be assigned to the container, and then no job could be executed
> on the cluster. Right?****
>
> Thanks!****
>
> ** **
>