Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HDFS, mail # user - RE: How to shuffle (Key,Value) pair from mapper to multiple reducer


Copy link to this message
-
Re: How to shuffle (Key,Value) pair from mapper to multiple reducer
Karthik Kambatla 2013-03-13, 15:59
How about sending <0,x> to 0 and <1,x> to 1; reduce 0 can act based on the
value of x?

On Wed, Mar 13, 2013 at 2:29 AM, Vikas Jadhav <[EMAIL PROTECTED]>wrote:

> Hello I am not talking about custom partioner(custom partitioner is
> involved but i want to write same pair for more number times)
> i want it go to two reducer.
> for example i have partioning attribute two dimensional
> <x1,x2>
>
> singatue     reduce
> <0,0>         0
> <0,1>         1
> <1,0>         2
> <1,1>         3
>
> for <1,0>   it will goto reducer
> for <1,null> it should goto to reducer 2 and 3
> for <0,null> it should goto reducer 0 and 1
>
> On Wed, Mar 13, 2013 at 2:32 PM, Viral Bajaria <[EMAIL PROTECTED]>wrote:
>
>> Do you want the pair to go to both reducers or do you want it to go to
>> only one but in a random fashion ?
>>
>> AFAIK, 1st is not possible. Someone on the list can correct if I am wrong.
>> 2nd is possible by just implementing your own partitioner which
>> randomizes where each key goes (not sure what you gain by that).
>>
>>
>> On Wed, Mar 13, 2013 at 1:59 AM, Vikas Jadhav <[EMAIL PROTECTED]>wrote:
>>
>>>
>>> Hi
>>> I am specifying requirement again with example.
>>>
>>>
>>>
>>> I have use case where i need to shufffle same (key,value) pair to
>>> multiple reducers
>>>
>>>
>>> For Example  we have pair  (1,"ABC") and two reducers (reducer0 and
>>> reducer1) are there then
>>>
>>> by default this pair will go to reduce1 (cause  (key % numOfReducer) >>> (1%2) )
>>>
>>>
>>> how i should shuffle this pair to both reducer.
>>>
>>> Also I willing to change the code of hadoop framework if Necessory.
>>>
>>>   Thank you
>>>
>>> On Wed, Mar 13, 2013 at 12:51 PM, feng lu <[EMAIL PROTECTED]> wrote:
>>>
>>>> Hi
>>>>
>>>> you can use Job#setNumReduceTasks(int tasks) method to set the number
>>>> of reducer to output.
>>>>
>>>>
>>>> On Wed, Mar 13, 2013 at 2:15 PM, Vikas Jadhav <[EMAIL PROTECTED]
>>>> > wrote:
>>>>
>>>>> Hello,
>>>>>
>>>>> As by default Hadoop framework can shuffle (key,value) pair to only
>>>>> one reducer
>>>>>
>>>>> I have use case where i need to shufffle same (key,value) pair to
>>>>> multiple reducers
>>>>>
>>>>> Also I  willing to change the code of hadoop framework if Necessory.
>>>>>
>>>>>
>>>>> Thank you
>>>>>
>>>>> --
>>>>> *
>>>>> *
>>>>> *
>>>>>
>>>>> Thanx and Regards*
>>>>> * Vikas Jadhav*
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Don't Grow Old, Grow Up... :-)
>>>>
>>>
>>>
>>>
>>> --
>>> *
>>> *
>>> *
>>>
>>> Thanx and Regards*
>>> * Vikas Jadhav*
>>>
>>
>>
>
>
> --
> *
> *
> *
>
> Thanx and Regards*
> * Vikas Jadhav*
>