Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
MapReduce, mail # user - Re: How to shuffle (Key,Value) pair from mapper to multiple reducer


+
feng lu 2013-03-13, 07:21
+
Vikas Jadhav 2013-03-13, 08:59
+
Viral Bajaria 2013-03-13, 09:02
Copy link to this message
-
Re: How to shuffle (Key,Value) pair from mapper to multiple reducer
Vikas Jadhav 2013-03-13, 09:29
Hello I am not talking about custom partioner(custom partitioner is
involved but i want to write same pair for more number times)
i want it go to two reducer.
for example i have partioning attribute two dimensional
<x1,x2>

singatue     reduce
<0,0>         0
<0,1>         1
<1,0>         2
<1,1>         3

for <1,0>   it will goto reducer
for <1,null> it should goto to reducer 2 and 3
for <0,null> it should goto reducer 0 and 1

On Wed, Mar 13, 2013 at 2:32 PM, Viral Bajaria <[EMAIL PROTECTED]>wrote:

> Do you want the pair to go to both reducers or do you want it to go to
> only one but in a random fashion ?
>
> AFAIK, 1st is not possible. Someone on the list can correct if I am wrong.
> 2nd is possible by just implementing your own partitioner which randomizes
> where each key goes (not sure what you gain by that).
>
>
> On Wed, Mar 13, 2013 at 1:59 AM, Vikas Jadhav <[EMAIL PROTECTED]>wrote:
>
>>
>> Hi
>> I am specifying requirement again with example.
>>
>>
>>
>> I have use case where i need to shufffle same (key,value) pair to
>> multiple reducers
>>
>>
>> For Example  we have pair  (1,"ABC") and two reducers (reducer0 and
>> reducer1) are there then
>>
>> by default this pair will go to reduce1 (cause  (key % numOfReducer) >> (1%2) )
>>
>>
>> how i should shuffle this pair to both reducer.
>>
>> Also I willing to change the code of hadoop framework if Necessory.
>>
>>   Thank you
>>
>> On Wed, Mar 13, 2013 at 12:51 PM, feng lu <[EMAIL PROTECTED]> wrote:
>>
>>> Hi
>>>
>>> you can use Job#setNumReduceTasks(int tasks) method to set the number of
>>> reducer to output.
>>>
>>>
>>> On Wed, Mar 13, 2013 at 2:15 PM, Vikas Jadhav <[EMAIL PROTECTED]>wrote:
>>>
>>>> Hello,
>>>>
>>>> As by default Hadoop framework can shuffle (key,value) pair to only one
>>>> reducer
>>>>
>>>> I have use case where i need to shufffle same (key,value) pair to
>>>> multiple reducers
>>>>
>>>> Also I  willing to change the code of hadoop framework if Necessory.
>>>>
>>>>
>>>> Thank you
>>>>
>>>> --
>>>> *
>>>> *
>>>> *
>>>>
>>>> Thanx and Regards*
>>>> * Vikas Jadhav*
>>>>
>>>
>>>
>>>
>>> --
>>> Don't Grow Old, Grow Up... :-)
>>>
>>
>>
>>
>> --
>> *
>> *
>> *
>>
>> Thanx and Regards*
>> * Vikas Jadhav*
>>
>
>
--
*
*
*

Thanx and Regards*
* Vikas Jadhav*
+
Artem Onuchin 2013-03-13, 09:12
+
samir das mohapatra 2013-03-13, 09:02