Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
MapReduce, mail # user - Does mapred.local.dir is important factor in reducer side?


+
Majid Azimi 2012-12-31, 16:14
+
Harsh J 2012-12-31, 19:25
+
Ted Dunning 2012-12-31, 19:17
Copy link to this message
-
Re: Does mapred.local.dir is important factor in reducer side?
Harsh J 2012-12-31, 19:28
Ted,

Do note that the local directory configs accept URIs in 2.x releases,
allowing users to plug alternative filesystems if they wanted to.

On Tue, Jan 1, 2013 at 12:47 AM, Ted Dunning <[EMAIL PROTECTED]> wrote:
> Hadoop, The Definitive Guide is only talking about Apache, CDH and
> Hortonworks here.
>
> The MapR distribution does not have this limitation and thus is one solution
> for this problem.
>
> Another solution is to do partial aggregates such as with a combiner.
>
>
> On Mon, Dec 31, 2012 at 8:14 AM, Majid Azimi <[EMAIL PROTECTED]>
> wrote:
>>
>> Hadoop the definitive guide says:
>> intermediate results on the mapper side is written to local disk at
>> mapred.local.dir location so if this location does not have enough space the
>> map will fail.
>>
>> I want to know if this is true on the reducer side. Output of all mappers
>> will merge at reducer side. In which location this merge happens? If that
>> location does not have enough space does reducer fail? What is the solution
>> for MapReduce jobs if intermediat results for some keys is more than local
>> disk of reducer?
>
>

--
Harsh J