-Re: Need info on mapred.child.java.opts, mapred.map.child.java.opts and mapred.reduce.child.java.opts
Harsh J 2013-03-08, 11:13
Its easier to understand if you know the history. First there was just
"mapred.child.java.opts", which controlled java options for both Map
and Reduce tasks (i.e. all tasks). Then there came a need for
task-specific java opts, so the project introduced
mapred.map.child.java.opts and mapred.reduce.child.java.opts, while
keeping around mapred.child.java.opts. Hence, if
mapred.map.child.java.opts is present, it is preferred over the
mapred.child.java.opts, likewise for mapred.reduce.child.java.opts vs.
mapred.child.java.opts. If neither of the specifics is present, we
look for mapred.child.java.opts.
P.s. Please do not cross-post to multiple email lists; it is a bad
practice and potentially spawns two different diverging conversation
threads on the same topic. This question is apt-enough for
[EMAIL PROTECTED] alone as it is not CDH specific, so I've moved
[EMAIL PROTECTED] to bcc.
On Fri, Mar 8, 2013 at 3:41 PM, Gaurav Dasgupta <[EMAIL PROTECTED]> wrote:
> While I was reading about the important Hadoop configuration properties, I
> came across a state of doubt regarding the Java heap space properties for
> the child tasks. According to my understanding, mapred.child.java.opts is
> the overall heap size allocated to any task (map or reduce). Then when we
> are setting mapred.map.child.java.opts and mapred.reduce.child.java.opts
> separately, are they overriding the mapred.child.java.opts?
> For example, if I have the following configuration:
> mapred.child.java.opts = -Xmx1g
> mapred.map.child.java.opts = -Xmx2g
> mapred.reduce.child.java.opts = -Xmx512m
> Then how exactly the memory allocation is getting distributed between map
> and reduce? My mapper gets more than the overall heap space as specified or
> it is restricted to 1g?
> Can some one help me understand this concept? Also, what are the other heap
> space related properties which we can use with the above and how?