Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
MapReduce >> mail # user >> Re: Parameter 'yarn.nodemanager.resource.cpu-cores' does not work


Copy link to this message
-
Re: Parameter 'yarn.nodemanager.resource.cpu-cores' does not work
Thanks, but seems it does not work for me.

My hadoop version is 'Hadoop 2.0.4-alpha', and seems it does not include a
class named
'org.apache.hadoop.yarn.util.resource.DominantResourceCalculator', so I
replaced it with
'org.apache.hadoop.yarn.util.LinuxResourceCalculatorPlugin'. My
configuration is below, and use the default values for other
LinuxContainerExecutor configurations. In my expectation, if I set the
value of 'yarn.nodemanager.resource.cpu-cores' to 0, no job could be
completed by nodemanager as the slaves do not have any cpu to use. But, in
fact, my job could be completed successfully. Why?

 <property>
    <name>yarn.nodemanager.resource.cpu-cores</name>
    <value>0</value>
  </property>

 <property>
    <name>yarn.nodemanager.vcores-pcores-ratio</name>
    <value>0</value>
  </property>

 <property>

<name>yarn.nodemanager.container-monitor.resource-calculator.class</name>
    <value>org.apache.hadoop.yarn.util.LinuxResourceCalculatorPlugin</value>
  </property>

 <property>
    <name>yarn.nodemanager.container-executor.class</name>

<value>org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor</value>
  </property>

2013/7/4 Chuan Liu <[EMAIL PROTECTED]>

>  I think you need to change the following configurations in yarn-site.xml
> to enable CPU resource limits.****
>
> ** **
>
> ‘yarn.nodemanager.container-monitor.resource-calculator.class’
> ‘org.apache.hadoop.yarn.util.resource.DominantResourceCalculator’****
>
> ** **
>
> ‘yarn.nodemanager.container-executor.class’****
>
> ‘org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor’****
>
> ** **
>
> Some LinuxContainerExecutor configurations:****
>
> yarn.nodemanager.linux-container-executor.path****
>
> yarn.nodemanager.linux-container-executor.resources-handler.class****
>
> yarn.nodemanager.linux-container-executor.cgroups.hierarchy****
>
> yarn.nodemanager.linux-container-executor.cgroups.mount****
>
> yarn.nodemanager.linux-container-executor.cgroups.mount-path****
>
> ** **
>
> -Chuan****
>
> ** **
>
> *From:* sam liu [mailto:[EMAIL PROTECTED]]
> *Sent:* Tuesday, July 02, 2013 8:33 PM
> *To:* [EMAIL PROTECTED]
> *Subject:* Parameter 'yarn.nodemanager.resource.cpu-cores' does not work**
> **
>
> ** **
>
> Hi,****
>
> With Hadoop 2.0.4-alpha, yarn.nodemanager.resource.cpu-cores does not work
> for me:****
>
> 1. The performance of running same terasort job do not change, even after
> increasing or decreasing the value of 'yarn.nodemanager.resource.cpu-cores'
> in yarn-site.xml and restart the yarn cluster.  ****
>
> 2. Even if I set the value of both
> 'yarn.nodemanager.resource.cpu-cores' and
> 'yarn.nodemanager.vcores-pcores-ratio' to 0, the MR job still could
> complete without any exception, but the expected behavior should be that no
> cpu could be assigned to the container, and then no job could be executed
> on the cluster. Right?****
>
> Thanks!****
>
> ** **
>
+
sam liu 2013-07-23, 03:01
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB