Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce >> mail # user >> How to Influence Reduce Task Location.


Copy link to this message
-
Re: How to Influence Reduce Task Location.
On 12/18/2010 12:43 PM, Jane Chen wrote:
> Hi All,
>
> Is there anyway to influence where a reduce task is run?  We have a case where we'd like to choose the host to run the reduce task based on the task's input key.
>
> Any suggestion is greatly appreciated.
>
> Thanks,
> Jane

We don't do exactly that, but we do something similar.

We don't make specific reducers run on specific hosts.  But we do
specifically shard our data - e.g., into 1024 shards - and we then run
1024 reducers, each of which runs on its correspondingly numbered shard
of the data.

DR
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB