-mapred.child.ulimit default value of null?
Keith Wiley 2011-01-21, 18:02
What is the interpretation of a default value for mapred.child.ulimit of null? Does that mean children are completely unlimited in their permission to use memory?
If I want to insure maximum memory availability to children, should I leave this null (on a cluster where that is the default), or should I set it to some fairly large number, say 8 GBs (bearing in mind that the parameter's units are KB)? Is null better given my desire here?
Also, what is a "child" in the context of this question (what does this parameter apply to)? For example, is a map task itself considered a "child" of the overall job or does this parameter not affect the map task itself but only child processes spawned *by* the map task?
In particular, how would this parameter (or any other relevant parameters you can think of, say, mapred.child.java.opts) affect the availability of memory to compiled C++ code (.so files in the distributed cache) called from a map task via JNI?
What I really want to do is insure that the native routines (C++ .so) called via JNI from a (Java) mapper have maximum memory availability/access on the node. Ostensibly, the native routines would be permitted to use the entire memory available on the machine.
Keith Wiley [EMAIL PROTECTED] www.keithwiley.com
"And what if we picked the wrong religion? Every week, we're just making God
madder and madder!"
-- Homer Simpson