Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # user >> Re: Container [pid=22885,containerID=container_1386156666044_0001_01_000013] is running beyond physical memory limits. Current usage: 1.0 GB of 1 GB physical memory used; 332.5 GB of 8 GB virtual memory used. Killing container.


Copy link to this message
-
Re: Container [pid=22885,containerID=container_1386156666044_0001_01_000013] is running beyond physical memory limits. Current usage: 1.0 GB of 1 GB physical memory used; 332.5 GB of 8 GB virtual memory used. Killing container.
Something looks really bad on your cluster. The JVM's heap size is 200MB
but its virtual memory has ballooned to a monstrous 332GB. Does that ring
any bell? Can you run regular java applications on this node? This doesn't
seem related to YARN per-se.

+Vinod
Hortonworks Inc.
http://hortonworks.com/
On Wed, Dec 4, 2013 at 5:16 AM, panfei <[EMAIL PROTECTED]> wrote:

>
>
> ---------- Forwarded message ----------
> From: panfei <[EMAIL PROTECTED]>
> Date: 2013/12/4
> Subject: Container
> [pid=22885,containerID=container_1386156666044_0001_01_000013] is running
> beyond physical memory limits. Current usage: 1.0 GB of 1 GB physical
> memory used; 332.5 GB of 8 GB virtual memory used. Killing container.
> To: CDH Users <[EMAIL PROTECTED]>
>
>
> Hi All:
>
> We are using CDH4.5 Hadoop for our production, when submit some (not all)
> jobs from hive, we get the following exception info , seems the physical
> memory and virtual memory both not enough for the job to run:
>
>
> Task with the most failures(4):
> -----
> Task ID:
>   task_1386156666044_0001_m_000000
>
> URL:
>
> http://namenode-1:8088/taskdetails.jsp?jobid=job_1386156666044_0001&tipid=task_1386156666044_0001_m_000000
> -----
> Diagnostic Messages for this Task:
> Container [pid=22885,containerID=container_1386156666044_0001_01_000013]
> is running beyond physical memory limits. Current usage: 1.0 GB of 1 GB
> physical memory used; 332.5 GB of 8 GB virtual memory used. Killing
> container.
> Dump of the process-tree for container_1386156666044_0001_01_000013 :
>         |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS)
> SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
>         |- 22885 22036 22885 22885 (java) 5414 108 356993519616 271953
> /usr/java/default/bin/java -Djava.net.preferIPv4Stack=true
> -Dhadoop.metrics.log.level=WARN -Xmx200m
> -Djava.io.tmpdir=/data/yarn/local/usercache/hive/appcache/application_1386156666044_0001/container_1386156666044_0001_01_000013/tmp
> -Dlog4j.configuration=container-log4j.properties
> -Dyarn.app.mapreduce.container.log.dir=/var/log/hadoop-yarn/containers/application_1386156666044_0001/container_1386156666044_0001_01_000013
> -Dyarn.app.mapreduce.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA
> org.apache.hadoop.mapred.YarnChild 192.168.101.55 60841
> attempt_1386156666044_0001_m_000000_3 13
>
> following is some of our configuration:
>
>   <property>
>     <name>yarn.nodemanager.resource.memory-mb</name>
>     <value>12288</value>
>   </property>
>
>   <property>
>     <name>yarn.nodemanager.vmem-pmem-ratio</name>
>     <value>8</value>
>   </property>
>
>   <property>
>     <name>yarn.nodemanager.vmem-check-enabled</name>
>     <value>false</value>
>   </property>
>
>   <property>
>     <name>yarn.nodemanager.resource.cpu-vcores</name>
>     <value>6</value>
>   </property>
>
> can you give me some advice? thanks a lot.
> --
> 不学习,不知道
>
>
>
> --
> 不学习,不知道
>

--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB