Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce >> mail # user >> how to start the customized Child JVM task process of m/r?


Copy link to this message
-
Re: how to start the customized Child JVM task process of m/r?
On Tue, Apr 26, 2011 at 6:02 AM, Juwei Shi <[EMAIL PROTECTED]> wrote:

As you know, Hadoop MapReduce start child JVM processes to run task. I want
> to start the process of m/r task myself thus I can pass some OS level
> parameters to the JVM process.
>

That currently isn't possible. The best that is available
is mapred.child.java.opts, which adds options to the child jvm.
Alternatively, you could fork a subprocess out of the map or reduce like
streaming does. Actually, I guess you could use streaming itself and get
control over how your code runs.

In MapReduce v2, which is described here
http://developer.yahoo.com/blogs/hadoop/posts/2011/02/mapreduce-nextgen/ ,
the user gets a lot more control over the task launching. In particular,
mapreduce becomes a user-land library controlled by the user.

-- Owen
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB