Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce >> mail # user >> how to start the customized Child JVM task process of m/r?


Copy link to this message
-
Re: how to start the customized Child JVM task process of m/r?
On Tue, Apr 26, 2011 at 6:02 AM, Juwei Shi <[EMAIL PROTECTED]> wrote:

As you know, Hadoop MapReduce start child JVM processes to run task. I want
> to start the process of m/r task myself thus I can pass some OS level
> parameters to the JVM process.
>

That currently isn't possible. The best that is available
is mapred.child.java.opts, which adds options to the child jvm.
Alternatively, you could fork a subprocess out of the map or reduce like
streaming does. Actually, I guess you could use streaming itself and get
control over how your code runs.

In MapReduce v2, which is described here
http://developer.yahoo.com/blogs/hadoop/posts/2011/02/mapreduce-nextgen/ ,
the user gets a lot more control over the task launching. In particular,
mapreduce becomes a user-land library controlled by the user.

-- Owen