Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce, mail # dev - oom during shuffle


Copy link to this message
-
oom during shuffle
Radim Kolar 2013-05-06, 02:35
i retested it and oom error is also in 2.0.3, but not in 1.x.

how much memory is needed for reducer during shuffle and sort? Peak
usage is memoryLimit or there have to be additional memory for
mapred.reduce.parallel.copies fetch buffers?

MergerManager: memoryLimit=326264416, maxSingleShuffleLimit=81566104,
mergeThreshold=215334528, ioSortFactor=10, memToMemMergeOutputsThreshold=10