Recently I have installed data nodes on Ubuntu 12.04 and observed failing
M/R jobs with errors like this:

Diagnostics report from attempt_1351154628597_0002_m_000000_0: Container
[pid=14529,containerID=container_1351154628597_0002_01_000002] is running
beyond virtual memory limits. Current usage: 124.4mb of 1.0gb physical
memory used; 2.1gb of 2.1gb virtual memory used. Killing container.
Dump of the process-tree for container_1351154628597_0002_01_000002 :
    |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS)
SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
    |- 14529 13550 14529 14529 (java) 678 18 2265411584 31856
/home/gd/gd/jdk1.6.0_35/bin/java -Djava.net.preferIPv4Stack=true
-Dhadoop.metrics.log.level=WARN -Xmx1000M -XX:MaxPermSize=512M
-Djava.io.tmpdir=/home/gd/gd/gi-de-nosql.cdh4-base/data/yarn/usercache/gd/appcache/application_1351154628597_0002/container_1351154628597_0002_01_000002/tmp
-Dlog4j.configuration=container-log4j.properties
-Dyarn.app.mapreduce.container.log.dir=/home/gd/gd/gi-de-nosql.cdh4-base/logs/application_1351154628597_0002/container_1351154628597_0002_01_000002
-Dyarn.app.mapreduce.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA
org.apache.hadoop.mapred.YarnChild 192.168.178.25 36183
attempt_1351154628597_0002_m_000000_0 2

I am using CDH4.0.1 (hadoop 2.0.0) with the Yarn M/R implementation on
Ubuntu 12.04 64Bit.

According to HADOOP-7154 making sure MALLOC_ARENA_MAX=1 (or 4) is exported
should fix the issue.

I tried the following:

Exporting the environment variable MALLOC_ARENA_MAX with value 1 in all
hadoop shell scrips (e.g. yarn-env.sh). Checking the launch_container.sh
script that Yarn creates I can tell that it indeed contains the line

export MALLOC_ARENA_MAX="1"

But still I am getting the error above.

In addition I tried adding

    <property>
       <name>mapred.child.env</name>
       <value>MALLOC_ARENA_MAX=1</value>
    </property>

to mapred-site.xml. But that didn't seem to fix it either.

Is there anything special that I need to configure on the server to make
the setting effective?

Any idea would be great!!

Thanks,
  Henning

Could not get it to make sense out of MALLOC_ARENA_MAX. No .bashrc etc.
no env script seemed to have any impact.

Made jobs work again by setting yarn.nodemanager.vmem-pmem-ratio=10. Now
they probably run with some obscene and unnecessary vmem allocation
(which I read does not come for free with the new malloc). What a crappy
situation (and change) :-(

Thanks,
   Henning

On 10/25/2012 11:47 AM, Henning Blohm wrote:
> Recently I have installed data nodes on Ubuntu 12.04 and observed
> failing M/R jobs with errors like this:
>
> Diagnostics report from attempt_1351154628597_0002_m_000000_0:
> Container
> [pid=14529,containerID=container_1351154628597_0002_01_000002] is
> running beyond virtual memory limits. Current usage: 124.4mb of 1.0gb
> physical memory used; 2.1gb of 2.1gb virtual memory used. Killing
> container.
> Dump of the process-tree for container_1351154628597_0002_01_000002 :
>     |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS)
> SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
>     |- 14529 13550 14529 14529 (java) 678 18 2265411584 31856
> /home/gd/gd/jdk1.6.0_35/bin/java -Djava.net.preferIPv4Stack=true
> -Dhadoop.metrics.log.level=WARN -Xmx1000M -XX:MaxPermSize=512M
> -Djava.io.tmpdir=/home/gd/gd/gi-de-nosql.cdh4-base/data/yarn/usercache/gd/appcache/application_1351154628597_0002/container_1351154628597_0002_01_000002/tmp
> -Dlog4j.configuration=container-log4j.properties
> -Dyarn.app.mapreduce.container.log.dir=/home/gd/gd/gi-de-nosql.cdh4-base/logs/application_1351154628597_0002/container_1351154628597_0002_01_000002
> -Dyarn.app.mapreduce.container.log.filesize=0
> -Dhadoop.root.logger=INFO,CLA org.apache.hadoop.mapred.YarnChild
> 192.168.178.25 36183 attempt_1351154628597_0002_m_000000_0 2
>
> I am using CDH4.0.1 (hadoop 2.0.0) with the Yarn M/R implementation on
> Ubuntu 12.04 64Bit.
>
> According to HADOOP-7154 making sure MALLOC_ARENA_MAX=1 (or 4) is
> exported should fix the issue.
>
> I tried the following:
>
> Exporting the environment variable MALLOC_ARENA_MAX with value 1 in
> all hadoop shell scrips (e.g. yarn-env.sh). Checking the
> launch_container.sh script that Yarn creates I can tell that it indeed
> contains the line
>
> export MALLOC_ARENA_MAX="1"
>
> But still I am getting the error above.
>
> In addition I tried adding
>
>     <property>
>        <name>mapred.child.env</name>
>        <value>MALLOC_ARENA_MAX=1</value>
>     </property>
>
> to mapred-site.xml. But that didn't seem to fix it either.
>
> Is there anything special that I need to configure on the server to
> make the setting effective?
>
> Any idea would be great!!
>
> Thanks,
>   Henning

1) Java uses different variables than 'malloc'. Look up 'Java garbage collection' to find out how it all works.

2) Is this a 32-bit kernel? Or Java version? Those top out at 2.1g address space. You need to run with a 64-bit kernel & Java to get real work done with Hadoop.

----- Original Message -----
| From: "Henning Blohm" <[EMAIL PROTECTED]>
| To: [EMAIL PROTECTED]
| Sent: Thursday, October 25, 2012 8:52:00 AM
| Subject: Re: Virtual memory problems on Ubuntu 12.04 (a.k.a. MALLOC_ARENA_MAX or HADOOP-7154)
|
| Could not get it to make sense out of MALLOC_ARENA_MAX. No .bashrc
| etc.
| no env script seemed to have any impact.
|
| Made jobs work again by setting yarn.nodemanager.vmem-pmem-ratio=10.
| Now
| they probably run with some obscene and unnecessary vmem allocation
| (which I read does not come for free with the new malloc). What a
| crappy
| situation (and change) :-(
|
| Thanks,
|    Henning
|
| On 10/25/2012 11:47 AM, Henning Blohm wrote:
| > Recently I have installed data nodes on Ubuntu 12.04 and observed
| > failing M/R jobs with errors like this:
| >
| > Diagnostics report from attempt_1351154628597_0002_m_000000_0:
| > Container
| > [pid=14529,containerID=container_1351154628597_0002_01_000002] is
| > running beyond virtual memory limits. Current usage: 124.4mb of
| > 1.0gb
| > physical memory used; 2.1gb of 2.1gb virtual memory used. Killing
| > container.
| > Dump of the process-tree for container_1351154628597_0002_01_000002
| > :
| >     |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS)
| > SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES)
| > FULL_CMD_LINE
| >     |- 14529 13550 14529 14529 (java) 678 18 2265411584 31856
| > /home/gd/gd/jdk1.6.0_35/bin/java -Djava.net.preferIPv4Stack=true
| > -Dhadoop.metrics.log.level=WARN -Xmx1000M -XX:MaxPermSize=512M
| > -Djava.io.tmpdir=/home/gd/gd/gi-de-nosql.cdh4-base/data/yarn/usercache/gd/appcache/application_1351154628597_0002/container_1351154628597_0002_01_000002/tmp
| > -Dlog4j.configuration=container-log4j.properties
| > -Dyarn.app.mapreduce.container.log.dir=/home/gd/gd/gi-de-nosql.cdh4-base/logs/application_1351154628597_0002/container_1351154628597_0002_01_000002
| > -Dyarn.app.mapreduce.container.log.filesize=0
| > -Dhadoop.root.logger=INFO,CLA org.apache.hadoop.mapred.YarnChild
| > 192.168.178.25 36183 attempt_1351154628597_0002_m_000000_0 2
| >
| > I am using CDH4.0.1 (hadoop 2.0.0) with the Yarn M/R implementation
| > on
| > Ubuntu 12.04 64Bit.
| >
| > According to HADOOP-7154 making sure MALLOC_ARENA_MAX=1 (or 4) is
| > exported should fix the issue.
| >
| > I tried the following:
| >
| > Exporting the environment variable MALLOC_ARENA_MAX with value 1 in
| > all hadoop shell scrips (e.g. yarn-env.sh). Checking the
| > launch_container.sh script that Yarn creates I can tell that it
| > indeed
| > contains the line
| >
| > export MALLOC_ARENA_MAX="1"
| >
| > But still I am getting the error above.
| >
| > In addition I tried adding
| >
| >     <property>
| >        <name>mapred.child.env</name>
| >        <value>MALLOC_ARENA_MAX=1</value>
| >     </property>
| >
| > to mapred-site.xml. But that didn't seem to fix it either.
| >
| > Is there anything special that I need to configure on the server to
| > make the setting effective?
| >
| > Any idea would be great!!
| >
| > Thanks,
| >   Henning
|
|

On 10/27/2012 03:45 AM, Lance Norskog wrote:
> 1) Java uses different variables than 'malloc'. Look up 'Java garbage collection' to find out how it all works.
What are you trying to say here? This is not a garbage collection
problem. It's a problem with the new arena collector in glibc that has
this impact on JVM executed multi-threaded apps (tempted to say look up
"MALLOC_ARENA_MAX JVM virtual memory" to find out how it all works).

>
> 2) Is this a 32-bit kernel? Or Java version? Those top out at 2.1g address space. You need to run with a 64-bit kernel & Java to get real work done with Hadoop.
As said in the original post: It's all 64-bit.

-- Henning
>
> ----- Original Message -----
> | From: "Henning Blohm" <[EMAIL PROTECTED]>
> | To: [EMAIL PROTECTED]
> | Sent: Thursday, October 25, 2012 8:52:00 AM
> | Subject: Re: Virtual memory problems on Ubuntu 12.04 (a.k.a. MALLOC_ARENA_MAX or HADOOP-7154)
> |
> | Could not get it to make sense out of MALLOC_ARENA_MAX. No .bashrc
> | etc.
> | no env script seemed to have any impact.
> |
> | Made jobs work again by setting yarn.nodemanager.vmem-pmem-ratio=10.
> | Now
> | they probably run with some obscene and unnecessary vmem allocation
> | (which I read does not come for free with the new malloc). What a
> | crappy
> | situation (and change) :-(
> |
> | Thanks,
> |    Henning
> |
> | On 10/25/2012 11:47 AM, Henning Blohm wrote:
> | > Recently I have installed data nodes on Ubuntu 12.04 and observed
> | > failing M/R jobs with errors like this:
> | >
> | > Diagnostics report from attempt_1351154628597_0002_m_000000_0:
> | > Container
> | > [pid=14529,containerID=container_1351154628597_0002_01_000002] is
> | > running beyond virtual memory limits. Current usage: 124.4mb of
> | > 1.0gb
> | > physical memory used; 2.1gb of 2.1gb virtual memory used. Killing
> | > container.
> | > Dump of the process-tree for container_1351154628597_0002_01_000002
> | > :
> | >     |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS)
> | > SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES)
> | > FULL_CMD_LINE
> | >     |- 14529 13550 14529 14529 (java) 678 18 2265411584 31856
> | > /home/gd/gd/jdk1.6.0_35/bin/java -Djava.net.preferIPv4Stack=true
> | > -Dhadoop.metrics.log.level=WARN -Xmx1000M -XX:MaxPermSize=512M
> | > -Djava.io.tmpdir=/home/gd/gd/gi-de-nosql.cdh4-base/data/yarn/usercache/gd/appcache/application_1351154628597_0002/container_1351154628597_0002_01_000002/tmp
> | > -Dlog4j.configuration=container-log4j.properties
> | > -Dyarn.app.mapreduce.container.log.dir=/home/gd/gd/gi-de-nosql.cdh4-base/logs/application_1351154628597_0002/container_1351154628597_0002_01_000002
> | > -Dyarn.app.mapreduce.container.log.filesize=0
> | > -Dhadoop.root.logger=INFO,CLA org.apache.hadoop.mapred.YarnChild
> | > 192.168.178.25 36183 attempt_1351154628597_0002_m_000000_0 2
> | >
> | > I am using CDH4.0.1 (hadoop 2.0.0) with the Yarn M/R implementation
> | > on
> | > Ubuntu 12.04 64Bit.
> | >
> | > According to HADOOP-7154 making sure MALLOC_ARENA_MAX=1 (or 4) is
> | > exported should fix the issue.
> | >
> | > I tried the following:
> | >
> | > Exporting the environment variable MALLOC_ARENA_MAX with value 1 in
> | > all hadoop shell scrips (e.g. yarn-env.sh). Checking the
> | > launch_container.sh script that Yarn creates I can tell that it
> | > indeed
> | > contains the line
> | >
> | > export MALLOC_ARENA_MAX="1"
> | >
> | > But still I am getting the error above.
> | >
> | > In addition I tried adding
> | >
> | >     <property>
> | >        <name>mapred.child.env</name>
> | >        <value>MALLOC_ARENA_MAX=1</value>
> | >     </property>
> | >
> | > to mapred-site.xml. But that didn't seem to fix it either.
> | >
> | > Is there anything special that I need to configure on the server to
> | > make the setting effective?
> | >
> | > Any idea would be great!!
> | >
> | > Thanks,
> | >   Henning
> |
> |