Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
HDFS >> mail # user >> OutOfMemory during Plain Java MapReduce


+
Christian Schneider 2013-03-07, 22:18
+
Christian Schneider 2013-03-07, 23:05
Copy link to this message
-
Re: OutOfMemory during Plain Java MapReduce
I posted this question to stackoverflow also:
http://stackoverflow.com/questions/15292061/how-to-implement-a-java-mapreduce-that-produce-output-values-large-then-the-maxi

Best Regards,
Christian.
2013/3/8 Christian Schneider <[EMAIL PROTECTED]>

> I had a look to the stacktrace and it says the problem is at the reducer:
> userSet.add(iterator.next().toString());
>
> Error: Java heap space
> attempt_201303072200_0016_r_000002_0: WARN : mapreduce.Counters - Group
> org.apache.hadoop.mapred.Task$Counter is deprecated. Use
> org.apache.hadoop.mapreduce.TaskCounter instead
> attempt_201303072200_0016_r_000002_0: WARN :
> org.apache.hadoop.conf.Configuration - session.id is deprecated. Instead,
> use dfs.metrics.session-id
> attempt_201303072200_0016_r_000002_0: WARN :
> org.apache.hadoop.conf.Configuration - slave.host.name is deprecated.
> Instead, use dfs.datanode.hostname
> attempt_201303072200_0016_r_000002_0: FATAL:
> org.apache.hadoop.mapred.Child - Error running child :
> java.lang.OutOfMemoryError: Java heap space
> attempt_201303072200_0016_r_000002_0: at
> java.util.Arrays.copyOfRange(Arrays.java:3209)
> attempt_201303072200_0016_r_000002_0: at
> java.lang.String.<init>(String.java:215)
> attempt_201303072200_0016_r_000002_0: at
> java.nio.HeapCharBuffer.toString(HeapCharBuffer.java:542)
> attempt_201303072200_0016_r_000002_0: at
> java.nio.CharBuffer.toString(CharBuffer.java:1157)
> attempt_201303072200_0016_r_000002_0: at
> org.apache.hadoop.io.Text.decode(Text.java:394)
> attempt_201303072200_0016_r_000002_0: at
> org.apache.hadoop.io.Text.decode(Text.java:371)
> attempt_201303072200_0016_r_000002_0: at
> org.apache.hadoop.io.Text.toString(Text.java:273)
> attempt_201303072200_0016_r_000002_0: at
> com.myCompany.UserToAppReducer.reduce(RankingReducer.java:21)
> attempt_201303072200_0016_r_000002_0: at com.myCompany.UserToAppReducer
> .reduce(RankingReducer.java:1)
> attempt_201303072200_0016_r_000002_0: at
> org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:164)
> attempt_201303072200_0016_r_000002_0: at
> org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:610)
> attempt_201303072200_0016_r_000002_0: at
> org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:444)
> attempt_201303072200_0016_r_000002_0: at
> org.apache.hadoop.mapred.Child$4.run(Child.java:268)
> attempt_201303072200_0016_r_000002_0: at
> java.security.AccessController.doPrivileged(Native Method)
> attempt_201303072200_0016_r_000002_0: at
> javax.security.auth.Subject.doAs(Subject.java:396)
> attempt_201303072200_0016_r_000002_0: at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
> attempt_201303072200_0016_r_000002_0: at
> org.apache.hadoop.mapred.Child.main(Child.java:262)
>
> But how to solve this?
>
>
> 2013/3/7 Christian Schneider <[EMAIL PROTECTED]>
>
>> Hi,
>> during the Reduce phase or afterwards (i don't really know how to debug
>> it) I get a heap out of Memory Exception.
>>
>> I guess this is because the value of the reduce task (a Custom Writable)
>> holds a List with a lot of user ids.
>> The Setup is quite simple. This are the related classes I used:
>>
>> //-----------------------------------------------
>> // The Reducer
>> // It just add all userIds of the Iterable to the UserSetWriteAble
>> //-----------------------------------------------
>> public class UserToAppReducer extends Reducer<Text, Text, Text,
>> UserSetWritable> {
>>
>> @Override
>>  protected void reduce(final Text appId, final Iterable<Text> userIds,
>> final Context context) throws IOException, InterruptedException  {
>>  final UserSetWritable userSet = new UserSetWritable();
>>
>> final Iterator<Text> iterator = userIds.iterator();
>>  while (iterator.hasNext()) {
>> userSet.add(iterator.next().toString());
>> }
>>
>> context.write(appId, userSet);
>> }
>> }
>>
>> //-----------------------------------------------
>> // The Custom Writable
>> // Needed to implement a own toString Method bring the output into the
+
Harsh J 2013-03-09, 00:55
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB