Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce, mail # user - How to Influence Reduce Task Location.


Copy link to this message
-
Re: How to Influence Reduce Task Location.
David Rosenstrauch 2010-12-20, 03:28
And, as a follow-up, yes, we use the partitioner class to achieve this.
  Our partioner runs a hashing algorithm which ensures that a given user
key will always map to a specific shard #.

DR

On 12/18/2010 01:16 PM, Hari Sreekumar wrote:
> Hi Jane,
>
>           The partitioner class can be used to achieve this. (
> http://hadoop.apache.org/mapreduce/docs/r0.21.0/api/org/apache/hadoop/mapreduce/Partitioner.html
> ).
>
> Thanks,
> Hari
>
> On Sat, Dec 18, 2010 at 11:13 PM, Jane Chen<[EMAIL PROTECTED]>wrote:
>
>> Hi All,
>>
>> Is there anyway to influence where a reduce task is run?  We have a case
>> where we'd like to choose the host to run the reduce task based on the
>> task's input key.
>>
>> Any suggestion is greatly appreciated.
>>
>> Thanks,
>> Jane