Hi All

I got the below exception, Is the issue related to
https://issues.apache.org/jira/browse/MAPREDUCE-1182 ?

Am using CDH3U1

2012-12-10 06:22:39,688 FATAL org.apache.hadoop.mapred.Task:
attempt_201211120903_9197_r_000024_0 : Map output copy failure :
java.lang.OutOfMemoryError: Java heap space
at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.shuffleInMemory(ReduceTask.java:1593)
at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.getMapOutput(ReduceTask.java:1453)
at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.copyOutput(ReduceTask.java:1302)
at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.run(ReduceTask.java:1234)

Cheers!
Manoj.

What was the job or query you were running?

Couple of suggestions:
1. Reduce data set size with job chaining
2. Increase Reduce task heap
3. If you are using Hive/Pig, you may want to tune your query.

-Bharath
________________________________
 From: Manoj Babu <[EMAIL PROTECTED]>
To: [EMAIL PROTECTED]
Sent: Monday, December 10, 2012 5:09 AM
Subject: Reg: Map output copy failure
 

Hi All

I got the below exception, Is the issue related to https://issues.apache.org/jira/browse/MAPREDUCE-1182 ?

Am using CDH3U1
2012-12-10 06:22:39,688 FATAL org.apache.hadoop.mapred.Task: attempt_201211120903_9197_r_000024_0 : Map output copy failure : java.lang.OutOfMemoryError: Java heap space at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.shuffleInMemory(ReduceTask.java:1593) at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.getMapOutput(ReduceTask.java:1453) at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.copyOutput(ReduceTask.java:1302) at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.run(ReduceTask.java:1234)Cheers!
Manoj.

In the ReduceTask.java having the below code

maxSize = (int)(conf.getInt("mapred.job.reduce.total.mem.bytes",
            (int)Math.min(Runtime.getRuntime().maxMemory(),
Integer.MAX_VALUE))
          * maxInMemCopyUse);

But in the patch
 maxSize = (long)Math.min(
             Runtime.getRuntime().maxMemory() * maxInMemCopyUse,
             Integer.MAX_VALUE);

Whether cdh3u1 not holding this fix?
Kindly Advice.

Cheers!
Manoj.

On Mon, Dec 10, 2012 at 6:39 PM, Manoj Babu <[EMAIL PROTECTED]> wrote:

> at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.copyOutput(ReduceTask.java:1302)
>
>

Please ignore the above comment.

Cheers!
Manoj.

On Mon, Dec 10, 2012 at 6:59 PM, Manoj Babu <[EMAIL PROTECTED]> wrote:

> In the ReduceTask.java having the below code
>
> maxSize = (int)(conf.getInt("mapred.job.reduce.total.mem.bytes",
>             (int)Math.min(Runtime.getRuntime().maxMemory(),
> Integer.MAX_VALUE))
>           * maxInMemCopyUse);
>
> But in the patch
>  maxSize = (long)Math.min(
>              Runtime.getRuntime().maxMemory() * maxInMemCopyUse,
>              Integer.MAX_VALUE);
>
> Whether cdh3u1 not holding this fix?
> Kindly Advice.
>
> Cheers!
> Manoj.
>
>
>
> On Mon, Dec 10, 2012 at 6:39 PM, Manoj Babu <[EMAIL PROTECTED]> wrote:
>
>> at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.copyOutput(ReduceTask.java:1302)
>>
>>
>