Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Hadoop, mail # user - Disable Sorting?


+
john smith 2011-09-10, 09:06
+
Arun C Murthy 2011-09-10, 18:48
+
Meng Mao 2011-09-10, 19:33
Copy link to this message
-
Re: Disable Sorting?
Owen O'Malley 2011-09-10, 20:33
On Sat, Sep 10, 2011 at 12:33 PM, Meng Mao <[EMAIL PROTECTED]> wrote:

> Is there a way to collate the possibly large number of map output files,
> though?
You can make fewer mappers by setting the mapred.min.split.size to define
the smallest input that will be given to a mapper.

There isn't currently a way of getting a collated, but unsorted list of
key/value pairs. For most applications, the in memory sort is fairly cheap
relative to the shuffle and other parts of the processing.

-- Owen
+
john smith 2011-09-11, 05:19
+
Arun C Murthy 2011-09-11, 01:33
+
john smith 2011-09-11, 07:43
+
Joey Echeverria 2011-09-11, 09:56