Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig >> mail # user >> StackOverflow exception in PIG


Copy link to this message
-
StackOverflow exception in PIG
Hi,

I have a pig script that uses a lot of OR conditions in the FILTER clause.
While the pig script works around with 500 'OR's, it throws a stack
overflow exception when i use more than that.

13/05/27 11:05:23 INFO mapReduceLayer.JobControlCompiler: Setting up single
store job
13/05/27 11:05:23 INFO data.SchemaTupleFrontend: Key [pig.schematuple] is
false, will not generate code.
13/05/27 11:05:23 INFO data.SchemaTupleFrontend: Starting process to move
generated code to distributed cacche
13/05/27 11:05:23 INFO data.SchemaTupleFrontend: Setting key
[pig.schematuple.classes] with classes to deserialize []
Exception in thread "main" java.lang.StackOverflowError
        at java.lang.Exception.<init>(Exception.java:102)
        at
java.lang.ReflectiveOperationException.<init>(ReflectiveOperationException.java:89)
        at
java.lang.reflect.InvocationTargetException.<init>(InvocationTargetException.java:72)
        at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:601)
        at
java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:975)
        at
java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1480)
        at
java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1416)
        at
java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1174)
        at
java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1528)
        at
java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1493)
        at
java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1416)
        at
java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1174)
        at
java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:346)
        at java.util.ArrayList.writeObject(ArrayList.java:710)
        at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:601)

How to resolve this in such scenario ? Should i just use multiple filters
and join all the relations to do what i intend to ?

--
Regards,
Praveen Bysani
http://www.praveenbysani.com