Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
HDFS >> mail # user >> Controlling on which node a reducer will be executed


+
Eduard Skaley 2012-08-27, 17:11
Copy link to this message
-
Re: Controlling on which node a reducer will be executed
Hi Eduard,

This isn't impossible, just unavailable at the moment. See
https://issues.apache.org/jira/browse/MAPREDUCE-199 if you want to
take a shot at implementing this. HBase would love to have this, I
think.

On Mon, Aug 27, 2012 at 10:41 PM, Eduard Skaley <[EMAIL PROTECTED]> wrote:
> Hi,
>
> i have a question concerning the execution of reducers.
> To use effectively the data locality of blocks in my use case i want to
> control on which node a reducer will be executed.
>
> In my scenario i have a chain of map-reduce jobs where each job will be
> executed by exactly N reducers.
> I want to achieve that for each job reducer 1 is executed on node 1,
> reducer 2 on node 2 and so on.
>
> Is it possible with a customization of the BlockPlacementPolicy in MRv2 ?
> Or is there some other way, maybe some indirect ? Or is it impossible ?
>
> Thanks for your help.
> Eduard

--
Harsh J
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB