Hi,

i have a question concerning the execution of reducers.
To use effectively the data locality of blocks in my use case i want to
control on which node a reducer will be executed.

In my scenario i have a chain of map-reduce jobs where each job will be
executed by exactly N reducers.
I want to achieve that for each job reducer 1 is executed on node 1,
reducer 2 on node 2 and so on.

Is it possible with a customization of the BlockPlacementPolicy in MRv2 ?
Or is there some other way, maybe some indirect ? Or is it impossible ?

Thanks for your help.
Eduard

Hi Eduard,

This isn't impossible, just unavailable at the moment. See
https://issues.apache.org/jira/browse/MAPREDUCE-199 if you want to
take a shot at implementing this. HBase would love to have this, I
think.

On Mon, Aug 27, 2012 at 10:41 PM, Eduard Skaley <[EMAIL PROTECTED]> wrote:
> Hi,
>
> i have a question concerning the execution of reducers.
> To use effectively the data locality of blocks in my use case i want to
> control on which node a reducer will be executed.
>
> In my scenario i have a chain of map-reduce jobs where each job will be
> executed by exactly N reducers.
> I want to achieve that for each job reducer 1 is executed on node 1,
> reducer 2 on node 2 and so on.
>
> Is it possible with a customization of the BlockPlacementPolicy in MRv2 ?
> Or is there some other way, maybe some indirect ? Or is it impossible ?
>
> Thanks for your help.
> Eduard

--
Harsh J