A simple fix, aside of applying a possible combiner, would be to raise
the number of reducers by a small margin such that per-partition data
falls in size.
On Mon, Dec 31, 2012 at 9:44 PM, Majid Azimi <[EMAIL PROTECTED]> wrote:
> hi guys,
> Hadoop the definitive guide says:
> intermediate results on the mapper side is written to local disk at
> mapred.local.dir location so if this location does not have enough space the
> map will fail.
> I want to know if this is true on the reducer side. Output of all mappers
> will merge at reducer side. In which location this merge happens? If that
> location does not have enough space does reducer fail? What is the solution
> for MapReduce jobs if intermediat results for some keys is more than local
> disk of reducer?