-Controlling on which node a reducer will be executed
Eduard Skaley 2012-08-27, 17:11
i have a question concerning the execution of reducers.
To use effectively the data locality of blocks in my use case i want to
control on which node a reducer will be executed.
In my scenario i have a chain of map-reduce jobs where each job will be
executed by exactly N reducers.
I want to achieve that for each job reducer 1 is executed on node 1,
reducer 2 on node 2 and so on.
Is it possible with a customization of the BlockPlacementPolicy in MRv2 ?
Or is there some other way, maybe some indirect ? Or is it impossible ?
Thanks for your help.
Harsh J 2012-08-27, 17:20