-
Re: Deserialization error when using Jython UDF in Pig 0.10 scriptCheolsoo Park 2012-06-19, 23:30
Hi,
- ERROR 2997: Unable to recreate exception from backed error:
> java.io.IOException: Deserialization error: could not instantiate
> 'org.apache.pig.scripting.jython.JythonFunction' with arguments
'[/transfer/marek/poc123-code/poc123-udf.py, parse_time_local]'
This is a bug in Pig-0.10. Please refer to PIg-2761 and its related jiras:
https://issues.apache.org/jira/browse/PIG-2761
https://issues.apache.org/jira/browse/PIG-2745
Thanks,
Cheolsoo
On Tue, Jun 19, 2012 at 4:28 PM, Chris Diehl <[EMAIL PROTECTED]> wrote:
> Hi Marek,
>
> I wish I could offer a solution. I tried all sorts of permutations and to
> no avail. I retreated back to our base install of Pig 0.8 where Jython UDFs
> work for me.
> Here's a list of things I've tried:
>
> 1) Setting JYTHON_HOME to the Jython path
> 2) Setting PIG_CLASSPATH to <Jython path>/jython.jar
> 3) Copying jython.jar to <Pig path>/lib
> 4) Using Jython 2.5.0 and 2.5.2
>
> I can't afford to burn any more time right now looking into this. I'm
> hoping the problem can be uncovered. Jython UDF support is critical for my
> productivity.
>
> Chris
>
> On Tue, Jun 19, 2012 at 2:40 PM, Marek Dudek <[EMAIL PROTECTED]
> >wrote:
>
> > Chris,
> >
> > I can confirm the problem; just run into it myself :(
> >
> > In my case I was able to run my code succesfully on MapR cluster (hadoop
> > 0.20.2), with Pig 0.9.1 and Jython 2.5.2.
> > Now, I am getting same "Deserialization error" after upgrading to Pig
> > 0.10.0
> > (see log snippets below).
> >
> > Any ideas/luck so far? Is there anything I can try/test to help?
> >
> > Marek
> >
> >
> >
> > -----------------
> > 2012-06-19 16:24:54,470 [main] INFO org.apache.pig.Main - Apache Pig
> > version 0.10.0 (r1328203) compiled Apr 20 2012, 05:54:07
> > 2012-06-19 16:24:54,470 [main] INFO org.apache.pig.Main - Logging error
> > messages to: /transfer/marek/poc123-code/pig_1340141094466.log
> > (...)
> > 2012-06-19 16:26:27,484 [main] ERROR
> > org.apache.pig.tools.pigstats.PigStatsUtil - 1 map reduce job(s) failed!
> > 2012-06-19 16:26:27,488 [main] INFO
> > org.apache.pig.tools.pigstats.SimplePigStats - Script Statistics:
> >
> > HadoopVersion PigVersion UserId StartedAt
> FinishedAt
> > Features
> > 0.20.2-dev 0.10.0 marek 2012-06-19 16:25:13
> 2012-06-19
> > 16:26:27 FILTER
> >
> > Failed!
> > (...)
> >
> > 2012-06-19 16:26:27,488 [main] INFO
> >
> >
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLaunch
> > er - Failed!
> > 2012-06-19 16:26:27,544 [main] ERROR
> org.apache.pig.tools.grunt.GruntParser
> > - ERROR 2997: Unable to recreate exception from backed error:
> > java.io.IOException: Deserialization error: could not instantiate
> > 'org.apache.pig.scripting.jython.JythonFunction' with arguments
> > '[/transfer/marek/poc123-code/poc123-udf.py, parse_time_local]'
> > at
> >
> >
> org.apache.pig.impl.util.ObjectSerializer.deserialize(ObjectSerializer.java:
> > 55)
> > at
> >
> >
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBa
> > se.setup(PigGenericMapBase.java:177)
> > at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:142)
> > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:654)
> > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:325)
> > at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
> > at java.security.AccessController.doPrivileged(Native Method)
> > at javax.security.auth.Subject.doAs(Subject.java:396)
> > at
> >
> >
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.ja
> > va:1109)
> > at org.apache.hadoop.mapred.Child.main(Child.java:264)
> > Caused by: java.lang.RuntimeException: could not instantiate
> > 'org.apache.pig.scripting.jython.JythonFunction' with arguments
> > '[/transfer/marek/poc123-code/poc123-udf.py, parse_time_local]'
> > at
> >
> org.apache.pig.impl.PigContext.instantiateFuncFromSpec(PigContext.java:553)
- ERROR 2997: Unable to recreate exception from backed error:
> java.io.IOException: Deserialization error: could not instantiate
> 'org.apache.pig.scripting.jython.JythonFunction' with arguments
'[/transfer/marek/poc123-code/poc123-udf.py, parse_time_local]'
This is a bug in Pig-0.10. Please refer to PIg-2761 and its related jiras:
https://issues.apache.org/jira/browse/PIG-2761
https://issues.apache.org/jira/browse/PIG-2745
Thanks,
Cheolsoo
On Tue, Jun 19, 2012 at 4:28 PM, Chris Diehl <[EMAIL PROTECTED]> wrote:
> Hi Marek,
>
> I wish I could offer a solution. I tried all sorts of permutations and to
> no avail. I retreated back to our base install of Pig 0.8 where Jython UDFs
> work for me.
> Here's a list of things I've tried:
>
> 1) Setting JYTHON_HOME to the Jython path
> 2) Setting PIG_CLASSPATH to <Jython path>/jython.jar
> 3) Copying jython.jar to <Pig path>/lib
> 4) Using Jython 2.5.0 and 2.5.2
>
> I can't afford to burn any more time right now looking into this. I'm
> hoping the problem can be uncovered. Jython UDF support is critical for my
> productivity.
>
> Chris
>
> On Tue, Jun 19, 2012 at 2:40 PM, Marek Dudek <[EMAIL PROTECTED]
> >wrote:
>
> > Chris,
> >
> > I can confirm the problem; just run into it myself :(
> >
> > In my case I was able to run my code succesfully on MapR cluster (hadoop
> > 0.20.2), with Pig 0.9.1 and Jython 2.5.2.
> > Now, I am getting same "Deserialization error" after upgrading to Pig
> > 0.10.0
> > (see log snippets below).
> >
> > Any ideas/luck so far? Is there anything I can try/test to help?
> >
> > Marek
> >
> >
> >
> > -----------------
> > 2012-06-19 16:24:54,470 [main] INFO org.apache.pig.Main - Apache Pig
> > version 0.10.0 (r1328203) compiled Apr 20 2012, 05:54:07
> > 2012-06-19 16:24:54,470 [main] INFO org.apache.pig.Main - Logging error
> > messages to: /transfer/marek/poc123-code/pig_1340141094466.log
> > (...)
> > 2012-06-19 16:26:27,484 [main] ERROR
> > org.apache.pig.tools.pigstats.PigStatsUtil - 1 map reduce job(s) failed!
> > 2012-06-19 16:26:27,488 [main] INFO
> > org.apache.pig.tools.pigstats.SimplePigStats - Script Statistics:
> >
> > HadoopVersion PigVersion UserId StartedAt
> FinishedAt
> > Features
> > 0.20.2-dev 0.10.0 marek 2012-06-19 16:25:13
> 2012-06-19
> > 16:26:27 FILTER
> >
> > Failed!
> > (...)
> >
> > 2012-06-19 16:26:27,488 [main] INFO
> >
> >
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLaunch
> > er - Failed!
> > 2012-06-19 16:26:27,544 [main] ERROR
> org.apache.pig.tools.grunt.GruntParser
> > - ERROR 2997: Unable to recreate exception from backed error:
> > java.io.IOException: Deserialization error: could not instantiate
> > 'org.apache.pig.scripting.jython.JythonFunction' with arguments
> > '[/transfer/marek/poc123-code/poc123-udf.py, parse_time_local]'
> > at
> >
> >
> org.apache.pig.impl.util.ObjectSerializer.deserialize(ObjectSerializer.java:
> > 55)
> > at
> >
> >
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBa
> > se.setup(PigGenericMapBase.java:177)
> > at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:142)
> > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:654)
> > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:325)
> > at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
> > at java.security.AccessController.doPrivileged(Native Method)
> > at javax.security.auth.Subject.doAs(Subject.java:396)
> > at
> >
> >
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.ja
> > va:1109)
> > at org.apache.hadoop.mapred.Child.main(Child.java:264)
> > Caused by: java.lang.RuntimeException: could not instantiate
> > 'org.apache.pig.scripting.jython.JythonFunction' with arguments
> > '[/transfer/marek/poc123-code/poc123-udf.py, parse_time_local]'
> > at
> >
> org.apache.pig.impl.PigContext.instantiateFuncFromSpec(PigContext.java:553)
