Hi,

As you know, Hadoop MapReduce start child JVM processes to run task. I want
to start the process of m/r task myself thus I can pass some OS level
parameters to the JVM process.

For example, set the CPU affinity of the process:
taskset -c $processor java_process_for_the_m/r_task_command

For another example, bind local memory to the processor running the process
using NUMA
numactl -membind 0 java_process_for_the_m/r_task_command

How could I implement this? Should I change Hadoop script or source code?

Thanks in advanced!
-Juwei

On Tue, Apr 26, 2011 at 6:02 AM, Juwei Shi <[EMAIL PROTECTED]> wrote:

As you know, Hadoop MapReduce start child JVM processes to run task. I want
> to start the process of m/r task myself thus I can pass some OS level
> parameters to the JVM process.
>

That currently isn't possible. The best that is available
is mapred.child.java.opts, which adds options to the child jvm.
Alternatively, you could fork a subprocess out of the map or reduce like
streaming does. Actually, I guess you could use streaming itself and get
control over how your code runs.

In MapReduce v2, which is described here
http://developer.yahoo.com/blogs/hadoop/posts/2011/02/mapreduce-nextgen/ ,
the user gets a lot more control over the task launching. In particular,
mapreduce becomes a user-land library controlled by the user.

-- Owen

Owen, thanks a lot!

Is is possible to do this by changing some source code of current MR?

2011/4/26 Owen O'Malley <[EMAIL PROTECTED]>

> On Tue, Apr 26, 2011 at 6:02 AM, Juwei Shi <[EMAIL PROTECTED]> wrote:
>
> As you know, Hadoop MapReduce start child JVM processes to run task. I want
>> to start the process of m/r task myself thus I can pass some OS level
>> parameters to the JVM process.
>>
>
> That currently isn't possible. The best that is available
> is mapred.child.java.opts, which adds options to the child jvm.
> Alternatively, you could fork a subprocess out of the map or reduce like
> streaming does. Actually, I guess you could use streaming itself and get
> control over how your code runs.
>
> In MapReduce v2, which is described here
> http://developer.yahoo.com/blogs/hadoop/posts/2011/02/mapreduce-nextgen/ ,
> the user gets a lot more control over the task launching. In particular,
> mapreduce becomes a user-land library controlled by the user.
>
> -- Owen
>

--
- Juwei Shi (史巨伟)

2011/4/26 Juwei Shi <[EMAIL PROTECTED]>

> Owen, thanks a lot!
>
> Is is possible to do this by changing some source code of current MR?
Of course. Look at launchTask in

http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20-security/src/mapred/org/apache/hadoop/mapred/TaskController.java

 Actually, since the TaskController is a plugin, you could make a new plugin
implementation that had the desired changes.

-- Owen

Owen, thanks a lot for your great advice! It really helps!

2011/4/27 Owen O'Malley <[EMAIL PROTECTED]>

>
>
> 2011/4/26 Juwei Shi <[EMAIL PROTECTED]>
>
>> Owen, thanks a lot!
>>
>> Is is possible to do this by changing some source code of current MR?
>
>
> Of course. Look at launchTask in
>
>
> http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20-security/src/mapred/org/apache/hadoop/mapred/TaskController.java
>
>  Actually, since the TaskController is a plugin, you could make a new
> plugin implementation that had the desired changes.
>
> -- Owen
>

--
- Juwei Shi (史巨伟)