Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Pig, mail # user - Correlation function out of memory error


+
Houssam H. 2013-02-21, 19:39
Copy link to this message
-
Re: Correlation function out of memory error
Johnny Zhang 2013-02-22, 22:18
Hi, Houssam:
What's the error in your pig log file? I were trying to reproduce it with
1000 rows, 500 columns.
A = load 'random.txt' using PigStorage(':') as
(f1:double,f2:double,.........,f500:double);
B = group A all;
D = foreach B generate group,COR(A.$0,A.$1,A.$2,A.$3,.......A.$499);
dump D;

The exception in pig log file is
Backend error message
---------------------
Error: java.lang.OutOfMemoryError: *GC overhead limit exceeded*
at java.lang.Double.valueOf(Double.java:492)
 at org.apache.pig.data.BinInterSedes.readDatum(BinInterSedes.java:390)
at org.apache.pig.data.BinInterSedes.readDatum(BinInterSedes.java:313)
 at
org.apache.pig.data.utils.SedesHelper.readGenericTuple(SedesHelper.java:144)
at org.apache.pig.data.BinInterSedes.readDatum(BinInterSedes.java:344)
 at org.apache.pig.data.BinInterSedes.readDatum(BinInterSedes.java:313)
at
org.apache.pig.data.utils.SedesHelper.readGenericTuple(SedesHelper.java:144)
 at org.apache.pig.data.BinInterSedes.readDatum(BinInterSedes.java:344)
at org.apache.pig.data.BinInterSedes.readDatum(BinInterSedes.java:313)
 at
org.apache.pig.data.InternalCachedBag$CachedBagIterator.hasNext(InternalCachedBag.java:208)
at org.apache.pig.builtin.COR.combine(COR.java:258)
 at org.apache.pig.builtin.COR$Intermed.exec(COR.java:171)
at org.apache.pig.builtin.COR$Intermed.exec(COR.java:164)
 at org.apache.pig.backend.hadoop.executionengine.physi

Backend error message
---------------------
Error: java.lang.OutOfMemoryError: Java heap space
 at java.lang.Double.valueOf(Double.java:492)
at org.apache.pig.data.BinInterSedes.readDatum(BinInterSedes.java:390)
 at org.apache.pig.data.BinInterSedes.readDatum(BinInterSedes.java:313)
at
org.apache.pig.data.utils.SedesHelper.readGenericTuple(SedesHelper.java:144)
 at org.apache.pig.data.BinInterSedes.readDatum(BinInterSedes.java:344)
at org.apache.pig.data.BinInterSedes.readDatum(BinInterSedes.java:313)
 at
org.apache.pig.data.utils.SedesHelper.readGenericTuple(SedesHelper.java:144)
at org.apache.pig.data.BinInterSedes.readDatum(BinInterSedes.java:344)
 at org.apache.pig.data.BinInterSedes.readDatum(BinInterSedes.java:313)
at
org.apache.pig.data.InternalCachedBag$CachedBagIterator.hasNext(InternalCachedBag.java:208)
 at org.apache.pig.builtin.COR.combine(COR.java:258)
at org.apache.pig.builtin.COR$Intermed.exec(COR.java:171)
 at org.apache.pig.builtin.COR$Intermed.exec(COR.java:164)
at org.apache.pig.backend.hadoop.executionengine.physicalLayer.ex

Backend error message
---------------------
Error: java.lang.OutOfMemoryError: GC overhead limit exceeded
at java.util.ArrayList.<init>(ArrayList.java:112)
 at org.apache.pig.data.DefaultTuple.<init>(DefaultTuple.java:67)
at org.apache.pig.data.BinSedesTuple.<init>(BinSedesTuple.java:67)
 at
org.apache.pig.data.BinSedesTupleFactory.newTuple(BinSedesTupleFactory.java:38)
at
org.apache.pig.data.utils.SedesHelper.readGenericTuple(SedesHelper.java:142)
 at org.apache.pig.data.BinInterSedes.readDatum(BinInterSedes.java:344)
at org.apache.pig.data.BinInterSedes.readDatum(BinInterSedes.java:313)
 at
org.apache.pig.data.utils.SedesHelper.readGenericTuple(SedesHelper.java:144)
at org.apache.pig.data.BinInterSedes.readDatum(BinInterSedes.java:344)
 at org.apache.pig.data.BinInterSedes.readDatum(BinInterSedes.java:313)
at
org.apache.pig.data.InternalCachedBag$CachedBagIterator.hasNext(InternalCachedBag.java:208)
 at org.apache.pig.builtin.COR.combine(COR.java:258)
at org.apache.pig.builtin.COR$Intermed.exec(COR.java:171)
 at org.apache.pig.builtin.COR$Inte

Backend error message
---------------------
Error: java.lang.OutOfMemoryError: Java heap space
 at
org.apache.pig.data.BinSedesTupleFactory.newTuple(BinSedesTupleFactory.java:38)
at
org.apache.pig.data.utils.SedesHelper.readGenericTuple(SedesHelper.java:142)
 at org.apache.pig.data.BinInterSedes.readDatum(BinInterSedes.java:344)
at org.apache.pig.data.BinInterSedes.readDatum(BinInterSedes.java:313)
 at
org.apache.pig.data.utils.SedesHelper.readGenericTuple(SedesHelper.java:144)
at org.apache.pig.data.BinInterSedes.readDatum(BinInterSedes.java:344)
 at org.apache.pig.data.BinInterSedes.readDatum(BinInterSedes.java:313)
at
org.apache.pig.data.InternalCachedBag$CachedBagIterator.hasNext(InternalCachedBag.java:208)
 at org.apache.pig.builtin.COR.combine(COR.java:258)
at org.apache.pig.builtin.COR$Intermed.exec(COR.java:171)
 at org.apache.pig.builtin.COR$Intermed.exec(COR.java:164)
at
org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POUserFunc.getNext(POUserFunc.java:337)
 at org.apache.pig.backend.hadoop.executionengin

Error message from task (map) task_201302211102_0561_m_000000
ERROR 6016: Out of memory.

org.apache.pig.backend.executionengine.ExecException: ERROR 6016: Out of
memory.
at java.lang.Double.valueOf(Double.java:492)
 at org.apache.pig.data.BinInterSedes.readDatum(BinInterSedes.java:390)
at org.apache.pig.data.BinInterSedes.readDatum(BinInterSedes.java:313)
 at
org.apache.pig.data.utils.SedesHelper.readGenericTuple(SedesHelper.java:144)
at org.apache.pig.data.BinInterSedes.readDatum(BinInterSedes.java:344)
 at org.apache.pig.data.BinInterSedes.readDatum(BinInterSedes.java:313)
at
org.apache.pig.data.utils.SedesHelper.readGenericTuple(SedesHelper.java:144)
 at org.apache.pig.data.BinInterSedes.readDatum(BinInterSedes.java:344)
at org.apache.pig.data.BinInterSedes.readDatum(BinInterSedes.java:313)
 at
org.apache.pig.data.InternalCachedBag$CachedBagIterator.hasNext(InternalCachedBag.java:208)
at org.apache.pig.builtin.COR.combine(COR.java:258)
 at org.apache.pig.builtin.COR$Intermed.exec(COR.java:171)
at org.apache.pig.builtin.COR$Intermed.exec(COR.java:164)
Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded
... 13 more
===============================================================================Error message from task (map) task_201302211102_0561_m_000000
ERROR 6016: Out of memory.

org.apache.pig.backend.executionengine.ExecException: ERROR 6016: Out of
memory.
 at java.lang.D
+
Johnny Zhang 2013-02-22, 23:04
+
Johnny Zhang 2013-02-23, 03:10
+
Houssam 2013-02-25, 08:30