the Child of our jobs are launched using user hdfs

we got OOM in some jobs when they try to mmap() a big file.

I manually checked the ulimit -v of hdfs on the slave nodes, they are
unlimited,

but when I launched a fake streaming job, whose mapper is simply bash -c '
ulimit -a ',
I found that it has a 1G virtual memory limit.
why is this??
looks like a Cloudera bug

we are using CDH3U3
Thanks
Yang

MR processes' Child JVMs run as the 'mapred' user in a non-secure
cluster (i.e. they run as the same user as the one that runs the
TaskTracker parent). Ensure the mapred user too has its default
virtual memory raised.

Also, for Child JVMs, the job config "mapred.child.ulimit" is applied,
so you will want to check that value's settings too.

In case you have a valid bug despite these two additional points you
need to consider, please report it onto
https://issues.cloudera.org/browse/DISTRO

(Btw, moved discussion to CDH users list [EMAIL PROTECTED], since
it may be CDH specific as you say)

On Thu, Jun 14, 2012 at 4:11 AM, Yang <[EMAIL PROTECTED]> wrote:
> the Child of our jobs are launched using user hdfs
>
> we got OOM in some jobs when they try to mmap() a big file.
>
> I manually checked the ulimit -v of hdfs on the slave nodes, they are
> unlimited,
>
> but when I launched a fake streaming job, whose mapper is simply bash -c '
> ulimit -a ',
> I found that it has a 1G virtual memory limit.
>
>
> why is this??
>
>
> looks like a Cloudera bug
>
> we are using CDH3U3
>
>
> Thanks
> Yang

--
Harsh J