Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce, mail # user - number of reducers


Copy link to this message
-
Re: number of reducers
jamal sasha 2012-11-20, 20:24
Awesome thanks . Works great now

On Tuesday, November 20, 2012, Bejoy KS <[EMAIL PROTECTED]> wrote:
> Hi Sasha
>
> By default the number or reducers are set to be 1. If you want more you
need to specify it as
>
> hadoop jar myJar.jar myClass -D mapred.reduce.tasks=20 ...
>
> Regards
> Bejoy KS
>
> Sent from handheld, please excuse typos.
> ________________________________
> From: jamal sasha <[EMAIL PROTECTED]>
> Date: Tue, 20 Nov 2012 14:38:54 -0500
> To: <[EMAIL PROTECTED]>
> ReplyTo: [EMAIL PROTECTED]
> Subject: number of reducers
>
>
> Hi,
>
>   I wrote a simple map reduce job in hadoop streaming.
>
>
>
> I am wondering if I am doing something wrong ..
>
> While number of mappers are projected to be around 1700.. reducers.. just
1?
>
> It’s couple of TB’s worth of data.
>
> What can I do to address this.
>
> Basically mapper looks like this
>
>
>
> For line in sys.stdin:
>
>     Print line
>
>
>
> Reducer
>
> For line in sys.stdin:
>
>     New_line = process_line(line)
>
>     Print new_line
>
>
>
>
>
> Thanks
>
>
>