-Re: Does mapred.local.dir is important factor in reducer side?
Harsh J 2012-12-31, 19:28
Do note that the local directory configs accept URIs in 2.x releases,
allowing users to plug alternative filesystems if they wanted to.
On Tue, Jan 1, 2013 at 12:47 AM, Ted Dunning <[EMAIL PROTECTED]> wrote:
> Hadoop, The Definitive Guide is only talking about Apache, CDH and
> Hortonworks here.
> The MapR distribution does not have this limitation and thus is one solution
> for this problem.
> Another solution is to do partial aggregates such as with a combiner.
> On Mon, Dec 31, 2012 at 8:14 AM, Majid Azimi <[EMAIL PROTECTED]>
>> Hadoop the definitive guide says:
>> intermediate results on the mapper side is written to local disk at
>> mapred.local.dir location so if this location does not have enough space the
>> map will fail.
>> I want to know if this is true on the reducer side. Output of all mappers
>> will merge at reducer side. In which location this merge happens? If that
>> location does not have enough space does reducer fail? What is the solution
>> for MapReduce jobs if intermediat results for some keys is more than local
>> disk of reducer?