Pig, mail # user - Re: CDH3 fail python udf

CDH3 fail python udf

Xiaomeng Wan 2011-03-31, 22:07

Hi,
We recently updated our hadoop from CDH2 to CDH3b4, and had problems
using some old python udfs. Runing in local mode still works, but in
hadoop mode, it gives errors like "could not instantiate
'org.apache.pig.scripting.jython.JythonFunction' with arguments...".
Anyone see similar error with python udf on this hadoop distribution?
We are using pig 0.8.0. Thanks!

Regards
Shawn

Re: CDH3 fail python udf

Aniket Mokashi 2011-03-31, 22:38

I think this might be because when you start in hadoop mode, your
classpath configuration does not have jython.jar. Can you put that
explicitly in classpath and check it out?

Thanks,
Aniket

On Thu, March 31, 2011 6:07 pm, Xiaomeng Wan wrote:
> Hi,
> We recently updated our hadoop from CDH2 to CDH3b4, and had problems
> using some old python udfs. Runing in local mode still works, but in hadoop
> mode, it gives errors like "could not instantiate
> 'org.apache.pig.scripting.jython.JythonFunction' with arguments...".
> Anyone see similar error with python udf on this hadoop distribution?
> We are using pig 0.8.0. Thanks!
>
>
> Regards
> Shawn
>
>
>

Re: CDH3 fail python udf

Xiaomeng Wan 2011-04-01, 16:06

Hi Aniket,

We put both jython.jar and myudf.py in classpath and also register
jython.jar in our pig script. It worked well before the upgrading,
only failed after.

Regards,
Shawn

On Thu, Mar 31, 2011 at 4:38 PM, Aniket Mokashi <[EMAIL PROTECTED]> wrote:
> I think this might be because when you start in hadoop mode, your
> classpath configuration does not have jython.jar. Can you put that
> explicitly in classpath and check it out?
>
> Thanks,
> Aniket
>
> On Thu, March 31, 2011 6:07 pm, Xiaomeng Wan wrote:
>> Hi,
>> We recently updated our hadoop from CDH2 to CDH3b4, and had problems
>> using some old python udfs. Runing in local mode still works, but in hadoop
>> mode, it gives errors like "could not instantiate
>> 'org.apache.pig.scripting.jython.JythonFunction' with arguments...".
>> Anyone see similar error with python udf on this hadoop distribution?
>> We are using pig 0.8.0. Thanks!
>>
>>
>> Regards
>> Shawn
>>
>>
>>
>
>
>

Re: CDH3 fail python udf

Aniket Mokashi 2011-04-01, 20:24

Hi Shawn,

Every time we throw an Exception with 'could not instantiate ..' error
message, we also pass down the real exception instance, this might be able
to point to the reason why we fail in this scenario.
Can you provide details of your exception message from the log?

The way this works is, when you register the myudf.py script we register
all the function names inside script to pig and when we use these
functions, we parse and construct them with JythonFunction constructor.

Thanks,
Aniket

On Fri, April 1, 2011 12:06 pm, Xiaomeng Wan wrote:
> Hi Aniket,
>
>
> We put both jython.jar and myudf.py in classpath and also register
> jython.jar in our pig script. It worked well before the upgrading, only
> failed after.
>
> Regards,
> Shawn
>
>
> On Thu, Mar 31, 2011 at 4:38 PM, Aniket Mokashi <[EMAIL PROTECTED]>
> wrote:
>
>> I think this might be because when you start in hadoop mode, your
>> classpath configuration does not have jython.jar. Can you put that
>> explicitly in classpath and check it out?
>>
>> Thanks,
>> Aniket
>>
>>
>> On Thu, March 31, 2011 6:07 pm, Xiaomeng Wan wrote:
>>
>>> Hi,
>>> We recently updated our hadoop from CDH2 to CDH3b4, and had problems
>>> using some old python udfs. Runing in local mode still works, but in
>>> hadoop mode, it gives errors like "could not instantiate
>>> 'org.apache.pig.scripting.jython.JythonFunction' with arguments...".
>>> Anyone see similar error with python udf on this hadoop distribution?
>>> We are using pig 0.8.0. Thanks!
>>>
>>>
>>>
>>> Regards
>>> Shawn
>>>
>>>
>>>
>>>
>>
>>
>>
>
>

Re: CDH3 fail python udf

Xiaomeng Wan 2011-04-01, 20:42

Hi Aniket,

Here is the stacktrace of the exception.

java.io.IOException: Deserialization error: could not instantiate
'org.apache.pig.scripting.jython.JythonFunction' with arguments
'[/home/shawn/TESS/code/mypyudfs.py, isStopWord]'
at org.apache.pig.impl.util.ObjectSerializer.deserialize(ObjectSerializer.java:55)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.setup(PigMapBase.java:151)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:142)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:646)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:322)
at org.apache.hadoop.mapred.Child$4.run(Child.java:251)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1115)
at org.apache.hadoop.mapred.Child.main(Child.java:245)
Caused by: java.lang.RuntimeException: could not instantiate
'org.apache.pig.scripting.jython.JythonFunction' with arguments
'[/home/shawn/TESS/code/mypyudfs.py, isStopWord]'
at org.apache.pig.impl.PigContext.instantiateFuncFromSpec(PigContext.java:502)
at org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POUserFunc.instantiateFunc(POUserFunc.java:109)
at org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POUserFunc.readObject(POUserFunc.java:451)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:974)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1849)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1753)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:351)
at java.util.ArrayList.readObject(ArrayList.java:593)
at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:974)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1849)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1753)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:351)
at java.util.HashMap.readObject(HashMap.java:1030)
at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:974)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1849)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1753)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1947)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1871)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1753)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1947)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1871)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1753)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:351)
at java.util.ArrayList.readObject(ArrayList.java:593)
at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:974)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1849)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1753)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1947)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1871)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1753)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:351)
at java.util.ArrayList.readObject(ArrayList.java:593)
at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:974)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1849)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1753)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:351)
at java.util.HashMap.readObject(HashMap.java:1030)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:974)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1849)

Re: CDH3 fail python udf

Aniket Mokashi 2011-04-01, 23:10

Hi Shawn,

I think this is more of CDH packaging problem than Pig problem. I suspect
this is related to Java versions of jython and other components.

You may look into
https://docs.cloudera.com/download/attachments/8784980/CDH3b3_Installation_Guide.pdf?version=1&modificationDate=1300229469101
for more details.

Thanks,
Aniket

On Fri, April 1, 2011 4:42 pm, Xiaomeng Wan wrote:
> Hi Aniket,
>
>
> Here is the stacktrace of the exception.
>
>
> java.io.IOException: Deserialization error: could not instantiate
> 'org.apache.pig.scripting.jython.JythonFunction' with arguments
> '[/home/shawn/TESS/code/mypyudfs.py, isStopWord]'
> at
> org.apache.pig.impl.util.ObjectSerializer.deserialize(ObjectSerializer.ja
> va:55)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.s
> etup(PigMapBase.java:151)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:142) at
> org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:646)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:322) at
> org.apache.hadoop.mapred.Child$4.run(Child.java:251)
> at java.security.AccessController.doPrivileged(Native Method) at
> javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation
> .java:1115)
> at org.apache.hadoop.mapred.Child.main(Child.java:245) Caused by:
> java.lang.RuntimeException: could not instantiate
> 'org.apache.pig.scripting.jython.JythonFunction' with arguments
> '[/home/shawn/TESS/code/mypyudfs.py, isStopWord]'
> at
> org.apache.pig.impl.PigContext.instantiateFuncFromSpec(PigContext.java:50
> 2)
> at
> org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOpe
> rators.POUserFunc.instantiateFunc(POUserFunc.java:109)
> at
> org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOpe
> rators.POUserFunc.readObject(POUserFunc.java:451)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java
> :39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorI
> mpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597) at
> java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:974)
> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1849) at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1753)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329) at
> java.io.ObjectInputStream.readObject(ObjectInputStream.java:351)
> at java.util.ArrayList.readObject(ArrayList.java:593) at
> sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source) at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorI
> mpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597) at
> java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:974)
> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1849) at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1753)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329) at
> java.io.ObjectInputStream.readObject(ObjectInputStream.java:351)
> at java.util.HashMap.readObject(HashMap.java:1030) at
> sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source) at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorI
> mpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597) at
> java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:974)
> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1849) at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1753)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329) at
> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1947)
> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1871) at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1753)

Re: CDH3 fail python udf

Xiaomeng Wan 2011-04-04, 15:36

Thanks! Aniket
I will look into it.

Regards,
Shawn

On Fri, Apr 1, 2011 at 5:10 PM, Aniket Mokashi <[EMAIL PROTECTED]> wrote:
> Hi Shawn,
>
> I think this is more of CDH packaging problem than Pig problem. I suspect
> this is related to Java versions of jython and other components.
>
> You may look into
> https://docs.cloudera.com/download/attachments/8784980/CDH3b3_Installation_Guide.pdf?version=1&modificationDate=1300229469101
> for more details.
>
> Thanks,
> Aniket
>
> On Fri, April 1, 2011 4:42 pm, Xiaomeng Wan wrote:
>> Hi Aniket,
>>
>>
>> Here is the stacktrace of the exception.
>>
>>
>> java.io.IOException: Deserialization error: could not instantiate
>> 'org.apache.pig.scripting.jython.JythonFunction' with arguments
>> '[/home/shawn/TESS/code/mypyudfs.py, isStopWord]'
>> at
>> org.apache.pig.impl.util.ObjectSerializer.deserialize(ObjectSerializer.ja
>> va:55)
>> at
>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.s
>> etup(PigMapBase.java:151)
>> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:142) at
>> org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:646)
>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:322) at
>> org.apache.hadoop.mapred.Child$4.run(Child.java:251)
>> at java.security.AccessController.doPrivileged(Native Method) at
>> javax.security.auth.Subject.doAs(Subject.java:396)
>> at
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation
>> .java:1115)
>> at org.apache.hadoop.mapred.Child.main(Child.java:245) Caused by:
>> java.lang.RuntimeException: could not instantiate
>> 'org.apache.pig.scripting.jython.JythonFunction' with arguments
>> '[/home/shawn/TESS/code/mypyudfs.py, isStopWord]'
>> at
>> org.apache.pig.impl.PigContext.instantiateFuncFromSpec(PigContext.java:50
>> 2)
>> at
>> org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOpe
>> rators.POUserFunc.instantiateFunc(POUserFunc.java:109)
>> at
>> org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOpe
>> rators.POUserFunc.readObject(POUserFunc.java:451)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java
>> :39)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorI
>> mpl.java:25)
>> at java.lang.reflect.Method.invoke(Method.java:597) at
>> java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:974)
>> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1849) at
>> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1753)
>> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329) at
>> java.io.ObjectInputStream.readObject(ObjectInputStream.java:351)
>> at java.util.ArrayList.readObject(ArrayList.java:593) at
>> sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source) at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorI
>> mpl.java:25)
>> at java.lang.reflect.Method.invoke(Method.java:597) at
>> java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:974)
>> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1849) at
>> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1753)
>> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329) at
>> java.io.ObjectInputStream.readObject(ObjectInputStream.java:351)
>> at java.util.HashMap.readObject(HashMap.java:1030) at
>> sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source) at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorI
>> mpl.java:25)
>> at java.lang.reflect.Method.invoke(Method.java:597) at
>> java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:974)
>> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1849) at
>> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1753)
>> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329) at
>