Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
MapReduce >> mail # user >> How can I sort a file with pairs <Key Value> in reverse order?


+
Pedro Sá da Costa 2013-06-11, 10:10
+
Bhasker Allene 2013-06-11, 11:57
+
Pedro Sá da Costa 2013-06-11, 12:05
+
Bhasker Allene 2013-06-11, 12:21
Copy link to this message
-
Re: How can I sort a file with pairs <Key Value> in reverse order?
Thanks for your help. Now I get it.
On 11 June 2013 14:21, Bhasker Allene <[EMAIL PROTECTED]> wrote:

>  Mapper Input
>
> 7 vim
> 2 emacs
> 9 firefox
>
> Mapper output ( new key = Integer.MAX_VALUE - key value)
> 2147483640 vim
> 2147483645 emacs
> 2147483636 firefox
>
> Note :Integer.MAX_VALUE is 2147483647 (which would be 2^31 - 1)
>
> Hadoop will sort the records for you.
> If you are using single reducer, reducer input would be
>
> 2147483636 firefox
> 2147483640 vim
> 2147483645 emacs
>
> reducer output ( this time subtract key from Integer.MAX_VALUE to get
> back original value)
> 9 firefox
> 7 vim
> 2 emacs
>
>
>  On 11/06/2013 13:05, Pedro Sá da Costa wrote:
>
> Even with  your answer I can't see how can I sort the data in reverse
> order. I forgot to mention that, the output result is produced by one
> reduce task. This means that, at any point of the execution of the job,
>  the data must be grouped and sorted in descendent order.
>
>
> On 11 June 2013 13:57, Bhasker Allene <[EMAIL PROTECTED]> wrote:
>
>>  One way to approach is emit Integer.MAX_VALUE - your key as output of
>> mapper.
>> Example
>> Mapper input
>>  7 vim
>> 2 emacs
>> 9 firefox
>>
>>  Mapper output
>> (Integer.MAX_VALUE - 7) vim
>> (Integer.MAX_VALUE - 2) emacs
>> (Integer.MAX_VALUE - 9) firefox
>>
>> If you need secondary sorting on second part, you have to use composite
>> key  and write  your own petitioner, comparator.
>>
>> Regards,
>> Bhasker
>>
>> On 11/06/2013 11:10, Pedro Sá da Costa wrote:
>>
>>  I created a MapReduce job example that that uses the sort mechanism of
>> hadoop to sort a file by the key in ascending order. This is an example of
>> the data:
>>
>>  7    vim
>>  2    emacs
>>  9    firefox
>>
>>  At the end, I get the result:
>>
>> 2    emacs
>> 7    vim
>>  9    firefox
>>
>>
>>
>> Now I want to sort in reverse order, for the result be:
>> 9    firefox
>> 7    vim
>> 2    emacs
>>
>>
>>
>> How can I sort a file with pairs <Key Value> in reverse order?
>>
>> --
>> Best regards,
>>
>>
>>
>>   --
>> Thanks & Regards,
>> Bhasker Allene
>>
>>
>
>
>  --
> Best regards,
>
>
> --
> Thanks & Regards,
> Bhasker Allene
>
>
--
Best regards,
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB