Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
MapReduce >> mail # dev >> oom during shuffle


Copy link to this message
-
oom during shuffle
i retested it and oom error is also in 2.0.3, but not in 1.x.

how much memory is needed for reducer during shuffle and sort? Peak
usage is memoryLimit or there have to be additional memory for
mapred.reduce.parallel.copies fetch buffers?

MergerManager: memoryLimit=326264416, maxSingleShuffleLimit=81566104,
mergeThreshold=215334528, ioSortFactor=10, memToMemMergeOutputsThreshold=10
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB