Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce >> mail # dev >> MapTask.java -sort part


Copy link to this message
-
Re: MapTask.java -sort part
Yes as Mariappan has already pointed out; the sort is pluggable so the
plug-in can also nullify it. See [1] for an example implementation of
the MapOutputCollector interface (class MapOutputCopier) to start
with. Per Mariappan, this does not do any sorting, but only merges.

[1] - http://svn.apache.org/repos/asf/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestMerge.java

On Tue, Apr 23, 2013 at 2:00 PM, Samaneh Shokuhi
<[EMAIL PROTECTED]> wrote:
> Hi Harsh,
> That is nice if branch-2 included that .Just one thing i want to be sure
> about that .In this patch MAPREDUCE-4807 (branch-2)  , sort in mapper is
> avoided ? Becouse i need a version of hadoop without sorting part in mapper
> and thats the reason i wanted to modify MapTask to disable the mapper sort
> .Do you think branch-2 is appropriate version for me ?
>
> Samaneh
>
>
> On Tue, Apr 23, 2013 at 10:01 AM, Harsh J <[EMAIL PROTECTED]> wrote:
>
>> Note that the Reducer also does a merge sort over the acquired map
>> data; but am not sure if you looked there as well.
>>
>> Mariappan's pointed JIRA (MAPREDUCE-4807) is already available in the
>> 2.0.3+ releases out today and in the current branch-2. It would be
>> simpler to reuse that than do these modifications.
>>
>> On Sun, Apr 21, 2013 at 5:24 PM, Samaneh Shokuhi
>> <[EMAIL PROTECTED]> wrote:
>> > Hi All,
>> > I have modified the MapTask.java file and tried to disable sort part for
>> > some reasons . I have removed this line
>> > *sorter.sort(MapOutputBuffer.this, kvstart, endPosition, reporter); *in
>> > sortAndSpill method but no changes seen in result. I have expected to get
>> > unsorted keys as mapper output but was not like that.
>> >
>> > My question is did i perform right action to disable sort or something
>> else
>> > needs to be done ?
>> >
>> > Samaneh
>>
>>
>>
>> --
>> Harsh J
>>

--
Harsh J
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB