Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Hadoop, mail # user - issue about Shuffled Maps in MR job summary


Copy link to this message
-
issue about Shuffled Maps in MR job summary
ch huang 2013-12-11, 05:25
hi,maillist:
           i run terasort with 16 reducers and 8 reducers,when i double
reducer number, the Shuffled maps is also double ,my question is the job
only run 20 map tasks (total input file is 10,and each file is 100M,my
block size is 64M,so split is 20) why i need shuffle 160 maps in 8 reducers
run and 320 maps in 16 reducers run?how to caculate the shuffle maps number?

16 reducer summary output:
 Shuffled Maps =320

 8 reducer summary output:

Shuffled Maps =160