Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Pig >> mail # user >> Correlation function out of memory error


+
Houssam H. 2013-02-21, 19:39
Copy link to this message
-
Re: Correlation function out of memory error
Hi, Houssam:
What's the error in your pig log file? I were trying to reproduce it with
1000 rows, 500 columns.
A = load 'random.txt' using PigStorage(':') as
(f1:double,f2:double,.........,f500:double);
B = group A all;
D = foreach B generate group,COR(A.$0,A.$1,A.$2,A.$3,.......A.$499);
dump D;

The exception in pig log file is
Backend error message
---------------------
Error: java.lang.OutOfMemoryError: *GC overhead limit exceeded*
at java.lang.Double.valueOf(Double.java:492)
 at org.apache.pig.data.BinInterSedes.readDatum(BinInterSedes.java:390)
at org.apache.pig.data.BinInterSedes.readDatum(BinInterSedes.java:313)
 at
org.apache.pig.data.utils.SedesHelper.readGenericTuple(SedesHelper.java:144)
at org.apache.pig.data.BinInterSedes.readDatum(BinInterSedes.java:344)
 at org.apache.pig.data.BinInterSedes.readDatum(BinInterSedes.java:313)
at
org.apache.pig.data.utils.SedesHelper.readGenericTuple(SedesHelper.java:144)
 at org.apache.pig.data.BinInterSedes.readDatum(BinInterSedes.java:344)
at org.apache.pig.data.BinInterSedes.readDatum(BinInterSedes.java:313)
 at
org.apache.pig.data.InternalCachedBag$CachedBagIterator.hasNext(InternalCachedBag.java:208)
at org.apache.pig.builtin.COR.combine(COR.java:258)
 at org.apache.pig.builtin.COR$Intermed.exec(COR.java:171)
at org.apache.pig.builtin.COR$Intermed.exec(COR.java:164)
 at org.apache.pig.backend.hadoop.executionengine.physi

Backend error message
---------------------
Error: java.lang.OutOfMemoryError: Java heap space
 at java.lang.Double.valueOf(Double.java:492)
at org.apache.pig.data.BinInterSedes.readDatum(BinInterSedes.java:390)
 at org.apache.pig.data.BinInterSedes.readDatum(BinInterSedes.java:313)
at
org.apache.pig.data.utils.SedesHelper.readGenericTuple(SedesHelper.java:144)
 at org.apache.pig.data.BinInterSedes.readDatum(BinInterSedes.java:344)
at org.apache.pig.data.BinInterSedes.readDatum(BinInterSedes.java:313)
 at
org.apache.pig.data.utils.SedesHelper.readGenericTuple(SedesHelper.java:144)
at org.apache.pig.data.BinInterSedes.readDatum(BinInterSedes.java:344)
 at org.apache.pig.data.BinInterSedes.readDatum(BinInterSedes.java:313)
at
org.apache.pig.data.InternalCachedBag$CachedBagIterator.hasNext(InternalCachedBag.java:208)
 at org.apache.pig.builtin.COR.combine(COR.java:258)
at org.apache.pig.builtin.COR$Intermed.exec(COR.java:171)
 at org.apache.pig.builtin.COR$Intermed.exec(COR.java:164)
at org.apache.pig.backend.hadoop.executionengine.physicalLayer.ex

Backend error message
---------------------
Error: java.lang.OutOfMemoryError: GC overhead limit exceeded
at java.util.ArrayList.<init>(ArrayList.java:112)
 at org.apache.pig.data.DefaultTuple.<init>(DefaultTuple.java:67)
at org.apache.pig.data.BinSedesTuple.<init>(BinSedesTuple.java:67)
 at
org.apache.pig.data.BinSedesTupleFactory.newTuple(BinSedesTupleFactory.java:38)
at
org.apache.pig.data.utils.SedesHelper.readGenericTuple(SedesHelper.java:142)
 at org.apache.pig.data.BinInterSedes.readDatum(BinInterSedes.java:344)
at org.apache.pig.data.BinInterSedes.readDatum(BinInterSedes.java:313)
 at
org.apache.pig.data.utils.SedesHelper.readGenericTuple(SedesHelper.java:144)
at org.apache.pig.data.BinInterSedes.readDatum(BinInterSedes.java:344)
 at org.apache.pig.data.BinInterSedes.readDatum(BinInterSedes.java:313)
at
org.apache.pig.data.InternalCachedBag$CachedBagIterator.hasNext(InternalCachedBag.java:208)
 at org.apache.pig.builtin.COR.combine(COR.java:258)
at org.apache.pig.builtin.COR$Intermed.exec(COR.java:171)
 at org.apache.pig.builtin.COR$Inte

Backend error message
---------------------
Error: java.lang.OutOfMemoryError: Java heap space
 at
org.apache.pig.data.BinSedesTupleFactory.newTuple(BinSedesTupleFactory.java:38)
at
org.apache.pig.data.utils.SedesHelper.readGenericTuple(SedesHelper.java:142)
 at org.apache.pig.data.BinInterSedes.readDatum(BinInterSedes.java:344)
at org.apache.pig.data.BinInterSedes.readDatum(BinInterSedes.java:313)
 at
org.apache.pig.data.utils.SedesHelper.readGenericTuple(SedesHelper.java:144)
at org.apache.pig.data.BinInterSedes.readDatum(BinInterSedes.java:344)
 at org.apache.pig.data.BinInterSedes.readDatum(BinInterSedes.java:313)
at
org.apache.pig.data.InternalCachedBag$CachedBagIterator.hasNext(InternalCachedBag.java:208)
 at org.apache.pig.builtin.COR.combine(COR.java:258)
at org.apache.pig.builtin.COR$Intermed.exec(COR.java:171)
 at org.apache.pig.builtin.COR$Intermed.exec(COR.java:164)
at
org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POUserFunc.getNext(POUserFunc.java:337)
 at org.apache.pig.backend.hadoop.executionengin

Error message from task (map) task_201302211102_0561_m_000000
ERROR 6016: Out of memory.

org.apache.pig.backend.executionengine.ExecException: ERROR 6016: Out of
memory.
at java.lang.Double.valueOf(Double.java:492)
 at org.apache.pig.data.BinInterSedes.readDatum(BinInterSedes.java:390)
at org.apache.pig.data.BinInterSedes.readDatum(BinInterSedes.java:313)
 at
org.apache.pig.data.utils.SedesHelper.readGenericTuple(SedesHelper.java:144)
at org.apache.pig.data.BinInterSedes.readDatum(BinInterSedes.java:344)
 at org.apache.pig.data.BinInterSedes.readDatum(BinInterSedes.java:313)
at
org.apache.pig.data.utils.SedesHelper.readGenericTuple(SedesHelper.java:144)
 at org.apache.pig.data.BinInterSedes.readDatum(BinInterSedes.java:344)
at org.apache.pig.data.BinInterSedes.readDatum(BinInterSedes.java:313)
 at
org.apache.pig.data.InternalCachedBag$CachedBagIterator.hasNext(InternalCachedBag.java:208)
at org.apache.pig.builtin.COR.combine(COR.java:258)
 at org.apache.pig.builtin.COR$Intermed.exec(COR.java:171)
at org.apache.pig.builtin.COR$Intermed.exec(COR.java:164)
Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded
... 13 more
===============================================================================Error message from task (map) task_201302211102_0561_m_000000
ERROR 6016: Out of memory.

org.apache.pig.backend.executionengine.ExecException: ERROR 6016: Out of
memory.
 at java.lang.D
+
Johnny Zhang 2013-02-22, 23:04
+
Johnny Zhang 2013-02-23, 03:10
+
Houssam 2013-02-25, 08:30
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB