Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce >> mail # user >> number of reducers

Copy link to this message
RE: number of reducers
I specify mine inside mapred-site.xml


From: Bejoy KS [mailto:[EMAIL PROTECTED]]
Sent: Tuesday, November 20, 2012 3:10 PM
Subject: Re: number of reducers

Hi Sasha

By default the number or reducers are set to be 1. If you want more you need to specify it as

hadoop jar myJar.jar myClass -D mapred.reduce.tasks=20 ...
Bejoy KS

Sent from handheld, please excuse typos.
From: jamal sasha <[EMAIL PROTECTED]>
Date: Tue, 20 Nov 2012 14:38:54 -0500
Subject: number of reducers


  I wrote a simple map reduce job in hadoop streaming.

I am wondering if I am doing something wrong ..

While number of mappers are projected to be around 1700.. reducers.. just 1?

It's couple of TB's worth of data.

What can I do to address this.

Basically mapper looks like this

For line in sys.stdin:

    Print line


For line in sys.stdin:

    New_line = process_line(line)

    Print new_line


NOTICE: This e-mail message and any attachments are confidential, subject to copyright and may be privileged. Any unauthorized use, copying or disclosure is prohibited. If you are not the intended recipient, please delete and contact the sender immediately. Please consider the environment before printing this e-mail. AVIS : le pr?sent courriel et toute pi?ce jointe qui l'accompagne sont confidentiels, prot?g?s par le droit d'auteur et peuvent ?tre couverts par le secret professionnel. Toute utilisation, copie ou divulgation non autoris?e est interdite. Si vous n'?tes pas le destinataire pr?vu de ce courriel, supprimez-le et contactez imm?diatement l'exp?diteur. Veuillez penser ? l'environnement avant d'imprimer le pr?sent courriel