Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Hadoop >> mail # user >> guessing number of reducers.


Copy link to this message
-
guessing number of reducers.
By default the number of reducers is set to 1..
Is there a good way to guess optimal number of reducers....
Or let's say i have tbs worth of data... mappers are of order 5000 or so...
But ultimately i am calculating , let's say, some average of whole data...
say average transaction occurring...
Now the output will be just one line in one "part"... rest of them will be
empty.So i am guessing i need loads of reducers but then most of them will
be empty but at the same time one reducer won't suffice..
What's the best way to solve this..
How to guess optimal number of reducers..
Thanks
+
Bejoy KS 2012-11-21, 16:50
+
Kartashov, Andy 2012-11-21, 17:49
+
Bejoy KS 2012-11-21, 18:21
+
jamal sasha 2012-11-21, 18:27
+
Mohammad Tariq 2012-11-21, 18:04
+
Manoj Babu 2012-11-21, 17:58
+
Bejoy KS 2012-11-21, 18:34
+
Manoj Babu 2012-11-22, 04:45
+
Kartashov, Andy 2012-11-21, 16:43
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB