Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
MapReduce, mail # user - Re: How to shuffle (Key,Value) pair from mapper to multiple reducer


+
feng lu 2013-03-13, 07:21
+
Vikas Jadhav 2013-03-13, 08:59
Copy link to this message
-
Re: How to shuffle (Key,Value) pair from mapper to multiple reducer
Viral Bajaria 2013-03-13, 09:02
Do you want the pair to go to both reducers or do you want it to go to only
one but in a random fashion ?

AFAIK, 1st is not possible. Someone on the list can correct if I am wrong.
2nd is possible by just implementing your own partitioner which randomizes
where each key goes (not sure what you gain by that).

On Wed, Mar 13, 2013 at 1:59 AM, Vikas Jadhav <[EMAIL PROTECTED]>wrote:

>
> Hi
> I am specifying requirement again with example.
>
>
>
> I have use case where i need to shufffle same (key,value) pair to multiple
> reducers
>
>
> For Example  we have pair  (1,"ABC") and two reducers (reducer0 and
> reducer1) are there then
>
> by default this pair will go to reduce1 (cause  (key % numOfReducer) > (1%2) )
>
>
> how i should shuffle this pair to both reducer.
>
> Also I willing to change the code of hadoop framework if Necessory.
>
>  Thank you
>
> On Wed, Mar 13, 2013 at 12:51 PM, feng lu <[EMAIL PROTECTED]> wrote:
>
>> Hi
>>
>> you can use Job#setNumReduceTasks(int tasks) method to set the number of
>> reducer to output.
>>
>>
>> On Wed, Mar 13, 2013 at 2:15 PM, Vikas Jadhav <[EMAIL PROTECTED]>wrote:
>>
>>> Hello,
>>>
>>> As by default Hadoop framework can shuffle (key,value) pair to only one
>>> reducer
>>>
>>> I have use case where i need to shufffle same (key,value) pair to
>>> multiple reducers
>>>
>>> Also I  willing to change the code of hadoop framework if Necessory.
>>>
>>>
>>> Thank you
>>>
>>> --
>>> *
>>> *
>>> *
>>>
>>> Thanx and Regards*
>>> * Vikas Jadhav*
>>>
>>
>>
>>
>> --
>> Don't Grow Old, Grow Up... :-)
>>
>
>
>
> --
> *
> *
> *
>
> Thanx and Regards*
> * Vikas Jadhav*
>
+
Vikas Jadhav 2013-03-13, 09:29
+
Artem Onuchin 2013-03-13, 09:12
+
samir das mohapatra 2013-03-13, 09:02