Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig >> mail # user >> Failing to make sense of an error.

Copy link to this message
Failing to make sense of an error.

I've been compiling some top 25 lists for the frequency with which
values appear in certain columns in a relation, and based on some of the
counts, am curious to see if some of the values occur particularly often
together. To this end, I've been running the following code:

abandonSpike4pvnUid = GROUP abandonSpike3 BY (pv_num, user_id);
abandonSpike5pvnUid = FOREACH abandonSpike4pvnUid GENERATE group.pv_num AS pvn, group.user_id AS uid, COUNT(abandonSpike3) AS session_count;
abandonSpike6pvnUid = FILTER abandonSpike5pvnUid BY (session_count > 20);
abandonSpike7pvnUid = ORDER abandonSpike6pvnUid BY session_count DESC;
abandonSpikePvnUid = LIMIT abandonSpike7pvnUid 25;
dump abandonSpikePvnUid;

Where abandonSpike3 has already been successfully grouped by single
columns to generate the existing top-25 lists.

What's odd is that I was getting the error:

2012-11-27 23:17:00,112 [main] ERROR org.apache.pig.tools.grunt.Grunt -
ERROR 2997: Unable to recreate exception from backed error: Error: Java
heap space

and figured that I'd probably made the groups too small (and therefore
the list of groups too big) for ORDER to run properly. My solution was
to insert the FILTER line above, but even trying to get the output of
that line without any ORDERing (i.e. dump abandonSpike6pvnUid), I'm
continuing to get the same error message.

It would seem that I'm misjudging what's over-filling the heap, and
would appreciate a tip on what else it might be. In case it helps, the
logfile.stack trace from my most recent attempt follows:

===============================================================================Backend error message
java.lang.Throwable: Child Error
        at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:242)
Caused by: java.io.IOException: Task process exit with nonzero status of 1.
        at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:229)

Backend error message
Error: GC overhead limit exceeded

Pig Stack Trace
ERROR 6015: During execution, encountered a Hadoop error.

org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable to open iterator for alias abandonSpike6pvnUid. Backend error : During execution, encountered a Hadoop error.
        at org.apache.pig.PigServer.openIterator(PigServer.java:753)
        at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:76)
        at org.apache.pig.Main.run(Main.java:455)
        at org.apache.pig.Main.main(Main.java:107)
Caused by: org.apache.pig.backend.executionengine.ExecException: ERROR
6015: During execution, encountered a Hadoop error.
        at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:242)
Caused by: java.lang.Throwable: Child Error
        ... 1 more
Caused by: java.io.IOException: Task process exit with nonzero status of 1.

Kris Coward http://unripe.melon.org/
GPG Fingerprint: 2BF3 957D 310A FEEC 4733  830E 21A4 05C7 1FEB 12B3