jamal sasha 2012-11-21, 16:38
Bejoy KS 2012-11-21, 16:50
Kartashov, Andy 2012-11-21, 17:49
Bejoy KS 2012-11-21, 18:21
jamal sasha 2012-11-21, 18:27
Mohammad Tariq 2012-11-21, 18:04
Manoj Babu 2012-11-21, 17:58
Bejoy KS 2012-11-21, 18:34
Manoj Babu 2012-11-22, 04:45
This is what I am using...
After you start your job, visit jobtracker's WebUI <ip-address>:50030
And look for Cluster summary. Reduce Task Capacity shall hint you what optimally set your number to. I could be wrong but it works for me. :)
Cluster Summary (Heap Size is *** MB/966.69 MB)
Running Map Tasks
Running Reduce Tasks
Occupied Map Slots
Occupied Reduce Slots
Reserved Map Slots
Reserved Reduce Slots
Map Task Capacity
Reduce Task Capacity
From: jamal sasha [mailto:[EMAIL PROTECTED]]
Sent: Wednesday, November 21, 2012 11:39 AM
To: [EMAIL PROTECTED]
Subject: guessing number of reducers.
By default the number of reducers is set to 1..
Is there a good way to guess optimal number of reducers....
Or let's say i have tbs worth of data... mappers are of order 5000 or so...
But ultimately i am calculating , let's say, some average of whole data... say average transaction occurring...
Now the output will be just one line in one "part"... rest of them will be empty.So i am guessing i need loads of reducers but then most of them will be empty but at the same time one reducer won't suffice..
What's the best way to solve this..
How to guess optimal number of reducers..
NOTICE: This e-mail message and any attachments are confidential, subject to copyright and may be privileged. Any unauthorized use, copying or disclosure is prohibited. If you are not the intended recipient, please delete and contact the sender immediately. Please consider the environment before printing this e-mail. AVIS : le pr?sent courriel et toute pi?ce jointe qui l'accompagne sont confidentiels, prot?g?s par le droit d'auteur et peuvent ?tre couverts par le secret professionnel. Toute utilisation, copie ou divulgation non autoris?e est interdite. Si vous n'?tes pas le destinataire pr?vu de ce courriel, supprimez-le et contactez imm?diatement l'exp?diteur. Veuillez penser ? l'environnement avant d'imprimer le pr?sent courriel