Hi,

I am trying to sort a list of numbers (one per line) using  hadoop
mapreduce.
Kindly suggest any reference and code.

How do I implement custom input format and recordreader so that both key and
value are the number?

I am using Hadoop 0.20.2

Thanks
Neil

--
Thanks and Regards
Neil
http://neilghosh.com

Start with these three wiki pages
http://hadoop.apache.org/common/docs/current/
then dig this
http://developer.yahoo.com/hadoop/tutorial/
after wards
http://oreilly.com/catalog/9780596521981

u sud be able to get a working solution in step 2 only.

regards
ranjib
On Mon, Sep 6, 2010 at 1:47 AM, Neil Ghosh <[EMAIL PROTECTED]> wrote:

> Hi,
>
> I am trying to sort a list of numbers (one per line) using  hadoop
> mapreduce.
> Kindly suggest any reference and code.
>
> How do I implement custom input format and recordreader so that both key
> and
> value are the number?
>
> I am using Hadoop 0.20.2
>
> Thanks
> Neil
>
> --
> Thanks and Regards
> Neil
> http://neilghosh.com
>

Hi,

On Mon, Sep 6, 2010 at 1:47 AM, Neil Ghosh <[EMAIL PROTECTED]> wrote:
> Hi,
>
> I am trying to sort a list of numbers (one per line) using  hadoop
> mapreduce.
> Kindly suggest any reference and code.
>
> How do I implement custom input format and recordreader so that both key and
> value are the number?
>
> I am using Hadoop 0.20.2
>

In the examples that come with Hadoop, you can find a 'sort' example.
The source is at o.a.h.examples.Sort.java. This should give you a good
starting point, IMO.

Thanks
Hemanth

> Thanks
> Neil
>
> --
> Thanks and Regards
> Neil
> http://neilghosh.com
>

The critical item is that your map's output key should be IntWritable
instead of Text. The default comparator for IntWritable will give you
properly sorted numbers. If you stringify the numbers and output them
as text, they'll get sorted as strings.

-- Owen

There is a call to seethe sort order as well, by changing the comparator.

James

Sent from my mobile. Please excuse the typos.

On 2010-09-06, at 12:06 AM, "Owen O'Malley" <[EMAIL PROTECTED]> wrote:

> The critical item is that your map's output key should be IntWritable
> instead of Text. The default comparator for IntWritable will give you
> properly sorted numbers. If you stringify the numbers and output them
> as text, they'll get sorted as strings.
>
> -- Owen

Thanks Owen, you were right.
I got the solution from Neal in IRC #hadoop

http://pastebin.com/DZKKhGSW

BTW thanks for giving the question specific response !

I already have the tutorials and links.

On Mon, Sep 6, 2010 at 11:35 AM, Owen O'Malley <[EMAIL PROTECTED]> wrote:

> The critical item is that your map's output key should be IntWritable
> instead of Text. The default comparator for IntWritable will give you
> properly sorted numbers. If you stringify the numbers and output them
> as text, they'll get sorted as strings.
>
> -- Owen
>

--
Thanks and Regards
Neil
http://neilghosh.com