Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
MapReduce >> mail # user >> Does mapred.local.dir is important factor in reducer side?


+
Majid Azimi 2012-12-31, 16:14
Copy link to this message
-
Re: Does mapred.local.dir is important factor in reducer side?
A simple fix, aside of applying a possible combiner, would be to raise
the number of reducers by a small margin such that per-partition data
falls in size.

On Mon, Dec 31, 2012 at 9:44 PM, Majid Azimi <[EMAIL PROTECTED]> wrote:
> hi guys,
>
> Hadoop the definitive guide says:
> intermediate results on the mapper side is written to local disk at
> mapred.local.dir location so if this location does not have enough space the
> map will fail.
>
> I want to know if this is true on the reducer side. Output of all mappers
> will merge at reducer side. In which location this merge happens? If that
> location does not have enough space does reducer fail? What is the solution
> for MapReduce jobs if intermediat results for some keys is more than local
> disk of reducer?

--
Harsh J
+
Ted Dunning 2012-12-31, 19:17
+
Harsh J 2012-12-31, 19:28
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB