Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
MapReduce >> mail # dev >> oom during shuffle

Copy link to this message
oom during shuffle
i retested it and oom error is also in 2.0.3, but not in 1.x.

how much memory is needed for reducer during shuffle and sort? Peak
usage is memoryLimit or there have to be additional memory for
mapred.reduce.parallel.copies fetch buffers?

MergerManager: memoryLimit=326264416, maxSingleShuffleLimit=81566104,
mergeThreshold=215334528, ioSortFactor=10, memToMemMergeOutputsThreshold=10