Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Hadoop >> mail # user >> Disable Sorting?


+
john smith 2011-09-10, 09:06
+
Arun C Murthy 2011-09-10, 18:48
+
Meng Mao 2011-09-10, 19:33
Copy link to this message
-
Re: Disable Sorting?
On Sat, Sep 10, 2011 at 12:33 PM, Meng Mao <[EMAIL PROTECTED]> wrote:

> Is there a way to collate the possibly large number of map output files,
> though?
You can make fewer mappers by setting the mapred.min.split.size to define
the smallest input that will be given to a mapper.

There isn't currently a way of getting a collated, but unsorted list of
key/value pairs. For most applications, the in memory sort is fairly cheap
relative to the shuffle and other parts of the processing.

-- Owen
+
john smith 2011-09-11, 05:19
+
Arun C Murthy 2011-09-11, 01:33
+
john smith 2011-09-11, 07:43
+
Joey Echeverria 2011-09-11, 09:56
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB