Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
MapReduce >> mail # user >> Re: How to shuffle (Key,Value) pair from mapper to multiple reducer


+
feng lu 2013-03-13, 07:21
+
Vikas Jadhav 2013-03-13, 08:59
Copy link to this message
-
Re: How to shuffle (Key,Value) pair from mapper to multiple reducer
Do you want the pair to go to both reducers or do you want it to go to only
one but in a random fashion ?

AFAIK, 1st is not possible. Someone on the list can correct if I am wrong.
2nd is possible by just implementing your own partitioner which randomizes
where each key goes (not sure what you gain by that).

On Wed, Mar 13, 2013 at 1:59 AM, Vikas Jadhav <[EMAIL PROTECTED]>wrote:

>
> Hi
> I am specifying requirement again with example.
>
>
>
> I have use case where i need to shufffle same (key,value) pair to multiple
> reducers
>
>
> For Example  we have pair  (1,"ABC") and two reducers (reducer0 and
> reducer1) are there then
>
> by default this pair will go to reduce1 (cause  (key % numOfReducer) > (1%2) )
>
>
> how i should shuffle this pair to both reducer.
>
> Also I willing to change the code of hadoop framework if Necessory.
>
>  Thank you
>
> On Wed, Mar 13, 2013 at 12:51 PM, feng lu <[EMAIL PROTECTED]> wrote:
>
>> Hi
>>
>> you can use Job#setNumReduceTasks(int tasks) method to set the number of
>> reducer to output.
>>
>>
>> On Wed, Mar 13, 2013 at 2:15 PM, Vikas Jadhav <[EMAIL PROTECTED]>wrote:
>>
>>> Hello,
>>>
>>> As by default Hadoop framework can shuffle (key,value) pair to only one
>>> reducer
>>>
>>> I have use case where i need to shufffle same (key,value) pair to
>>> multiple reducers
>>>
>>> Also I  willing to change the code of hadoop framework if Necessory.
>>>
>>>
>>> Thank you
>>>
>>> --
>>> *
>>> *
>>> *
>>>
>>> Thanx and Regards*
>>> * Vikas Jadhav*
>>>
>>
>>
>>
>> --
>> Don't Grow Old, Grow Up... :-)
>>
>
>
>
> --
> *
> *
> *
>
> Thanx and Regards*
> * Vikas Jadhav*
>
+
Vikas Jadhav 2013-03-13, 09:29
+
Artem Onuchin 2013-03-13, 09:12
+
samir das mohapatra 2013-03-13, 09:02
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB