Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
HDFS >> mail # user >> Controlling on which node a reducer will be executed


Copy link to this message
-
Controlling on which node a reducer will be executed
Hi,

i have a question concerning the execution of reducers.
To use effectively the data locality of blocks in my use case i want to
control on which node a reducer will be executed.

In my scenario i have a chain of map-reduce jobs where each job will be
executed by exactly N reducers.
I want to achieve that for each job reducer 1 is executed on node 1,
reducer 2 on node 2 and so on.

Is it possible with a customization of the BlockPlacementPolicy in MRv2 ?
Or is there some other way, maybe some indirect ? Or is it impossible ?

Thanks for your help.
Eduard
+
Harsh J 2012-08-27, 17:20