William 2010-11-23, 20:58
I think the most proper config parameter to try is io.sort.factor, which
affects disk spilling times on both map and reduce side. The default value
of this parameter is 10, try to enlarge it to 100 or more.
If the spilling on reduce side is still frequent you could try tuning
up mapred.job.shuffle.input.buffer.percent along with
mapred.child.java.opts, which may reduce disk spilling times in the shuffle
phase. The default value of mapred.job.shuffle.input.buffer.percent is 0.7,
with mapred.child.java.opts -Xmx200m by default.
Notice that increasing these values will also increase the memory cost, so
we need to make sure memory won't become the system bottleneck.
Hope this could help.
On 24 November 2010 04:58, William <[EMAIL PROTECTED]> wrote:
> We are currently modifying the configuration of our hadoop grid (250
> machines). The machines are homogeneous and the specs are
> dual quad core cpu 18Gb ram 8x1tb drives
> currently we have set this up -
> 8 reduce slots at 800mb
> 8 map slots at 800mb
> raised our io.sort.mb to 256mb
> we see a lot of spilling on both maps and reduces and I am wondering what
> other configs I should be looking into