Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce, mail # user - Does mapred.local.dir is important factor in reducer side?


Copy link to this message
-
Re: Does mapred.local.dir is important factor in reducer side?
Harsh J 2012-12-31, 19:25
A simple fix, aside of applying a possible combiner, would be to raise
the number of reducers by a small margin such that per-partition data
falls in size.

On Mon, Dec 31, 2012 at 9:44 PM, Majid Azimi <[EMAIL PROTECTED]> wrote:
> hi guys,
>
> Hadoop the definitive guide says:
> intermediate results on the mapper side is written to local disk at
> mapred.local.dir location so if this location does not have enough space the
> map will fail.
>
> I want to know if this is true on the reducer side. Output of all mappers
> will merge at reducer side. In which location this merge happens? If that
> location does not have enough space does reducer fail? What is the solution
> for MapReduce jobs if intermediat results for some keys is more than local
> disk of reducer?

--
Harsh J