Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HDFS >> mail # user >> Controlling on which node a reducer will be executed


Copy link to this message
-
Controlling on which node a reducer will be executed
Hi,

i have a question concerning the execution of reducers.
To use effectively the data locality of blocks in my use case i want to
control on which node a reducer will be executed.

In my scenario i have a chain of map-reduce jobs where each job will be
executed by exactly N reducers.
I want to achieve that for each job reducer 1 is executed on node 1,
reducer 2 on node 2 and so on.

Is it possible with a customization of the BlockPlacementPolicy in MRv2 ?
Or is there some other way, maybe some indirect ? Or is it impossible ?

Thanks for your help.
Eduard
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB