Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig >> mail # user >> StackOverflow exception in PIG


Copy link to this message
-
StackOverflow exception in PIG
Hi,

I have a pig script that uses a lot of OR conditions in the FILTER clause.
While the pig script works around with 500 'OR's, it throws a stack
overflow exception when i use more than that.

13/05/27 11:05:23 INFO mapReduceLayer.JobControlCompiler: Setting up single
store job
13/05/27 11:05:23 INFO data.SchemaTupleFrontend: Key [pig.schematuple] is
false, will not generate code.
13/05/27 11:05:23 INFO data.SchemaTupleFrontend: Starting process to move
generated code to distributed cacche
13/05/27 11:05:23 INFO data.SchemaTupleFrontend: Setting key
[pig.schematuple.classes] with classes to deserialize []
Exception in thread "main" java.lang.StackOverflowError
        at java.lang.Exception.<init>(Exception.java:102)
        at
java.lang.ReflectiveOperationException.<init>(ReflectiveOperationException.java:89)
        at
java.lang.reflect.InvocationTargetException.<init>(InvocationTargetException.java:72)
        at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:601)
        at
java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:975)
        at
java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1480)
        at
java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1416)
        at
java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1174)
        at
java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1528)
        at
java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1493)
        at
java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1416)
        at
java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1174)
        at
java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:346)
        at java.util.ArrayList.writeObject(ArrayList.java:710)
        at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:601)

How to resolve this in such scenario ? Should i just use multiple filters
and join all the relations to do what i intend to ?

--
Regards,
Praveen Bysani
http://www.praveenbysani.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB