Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce, mail # user - How to Influence Reduce Task Location.


Copy link to this message
-
Re: How to Influence Reduce Task Location.
David Rosenstrauch 2010-12-20, 03:26
On 12/18/2010 12:43 PM, Jane Chen wrote:
> Hi All,
>
> Is there anyway to influence where a reduce task is run?  We have a case where we'd like to choose the host to run the reduce task based on the task's input key.
>
> Any suggestion is greatly appreciated.
>
> Thanks,
> Jane

We don't do exactly that, but we do something similar.

We don't make specific reducers run on specific hosts.  But we do
specifically shard our data - e.g., into 1024 shards - and we then run
1024 reducers, each of which runs on its correspondingly numbered shard
of the data.

DR