Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce, mail # dev - MapTask.java -sort part


Copy link to this message
-
Re: MapTask.java -sort part
Harsh J 2013-04-23, 08:49
Yes as Mariappan has already pointed out; the sort is pluggable so the
plug-in can also nullify it. See [1] for an example implementation of
the MapOutputCollector interface (class MapOutputCopier) to start
with. Per Mariappan, this does not do any sorting, but only merges.

[1] - http://svn.apache.org/repos/asf/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestMerge.java

On Tue, Apr 23, 2013 at 2:00 PM, Samaneh Shokuhi
<[EMAIL PROTECTED]> wrote:
> Hi Harsh,
> That is nice if branch-2 included that .Just one thing i want to be sure
> about that .In this patch MAPREDUCE-4807 (branch-2)  , sort in mapper is
> avoided ? Becouse i need a version of hadoop without sorting part in mapper
> and thats the reason i wanted to modify MapTask to disable the mapper sort
> .Do you think branch-2 is appropriate version for me ?
>
> Samaneh
>
>
> On Tue, Apr 23, 2013 at 10:01 AM, Harsh J <[EMAIL PROTECTED]> wrote:
>
>> Note that the Reducer also does a merge sort over the acquired map
>> data; but am not sure if you looked there as well.
>>
>> Mariappan's pointed JIRA (MAPREDUCE-4807) is already available in the
>> 2.0.3+ releases out today and in the current branch-2. It would be
>> simpler to reuse that than do these modifications.
>>
>> On Sun, Apr 21, 2013 at 5:24 PM, Samaneh Shokuhi
>> <[EMAIL PROTECTED]> wrote:
>> > Hi All,
>> > I have modified the MapTask.java file and tried to disable sort part for
>> > some reasons . I have removed this line
>> > *sorter.sort(MapOutputBuffer.this, kvstart, endPosition, reporter); *in
>> > sortAndSpill method but no changes seen in result. I have expected to get
>> > unsorted keys as mapper output but was not like that.
>> >
>> > My question is did i perform right action to disable sort or something
>> else
>> > needs to be done ?
>> >
>> > Samaneh
>>
>>
>>
>> --
>> Harsh J
>>

--
Harsh J