Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce >> mail # user >> secondary sort - number of reducers


Copy link to this message
-
Re: secondary sort - number of reducers
No...partitionr decides which keys should go to which reducer...and
number of reducers you need to decide...No of reducers depends on
factors like number of key value pair, use case etc
Regards,
Som Shekhar Sharma
+91-8197243810
On Fri, Aug 30, 2013 at 5:54 AM, Adeel Qureshi <[EMAIL PROTECTED]> wrote:
> so it cant figure out an appropriate number of reducers as it does for
> mappers .. in my case hadoop is using 2100+ mappers and then only 1 reducer
> .. since im overriding the partitioner class shouldnt that decide how
> manyredeucers there should be based on how many different partition values
> being returned by the custom partiotioner
>
>
> On Thu, Aug 29, 2013 at 7:38 PM, Ian Wrigley <[EMAIL PROTECTED]> wrote:
>>
>> If you don't specify the number of Reducers, Hadoop will use the default
>> -- which, unless you've changed it, is 1.
>>
>> Regards
>>
>> Ian.
>>
>> On Aug 29, 2013, at 4:23 PM, Adeel Qureshi <[EMAIL PROTECTED]> wrote:
>>
>> I have implemented secondary sort in my MR job and for some reason if i
>> dont specify the number of reducers it uses 1 which doesnt seems right
>> because im working with 800M+ records and one reducer slows things down
>> significantly. Is this some kind of limitation with the secondary sort that
>> it has to use a single reducer .. that kind of would defeat the purpose of
>> having a scalable solution such as secondary sort. I would appreciate any
>> help.
>>
>> Thanks
>> Adeel
>>
>>
>>
>> ---
>> Ian Wrigley
>> Sr. Curriculum Manager
>> Cloudera, Inc
>> Cell: (323) 819 4075
>>
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB