Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig >> mail # user >> python modules


Copy link to this message
-
Re: python modules
Root cause for this problem was
https://issues.apache.org/jira/browse/MAPREDUCE-967 backward
incompatibility.

Hadoop unpacks job.jar and puts jobCacheDir on classpath and hence jython
is not able to find its jar (job.jar) on classpath (as its unpacked).

With MAPREDUCE-967, hadoop puts job.jar on classpath and hence jython finds
its jar (=job.jar) and updates the sys.path.

Stan, I think if you put jython.jar on classpath for TT, it will work. (you
have to restart TT). PIG-2010 is another possible choice.

Thanks,
Aniket

On Mon, Mar 12, 2012 at 1:16 AM, Aniket Mokashi <[EMAIL PROTECTED]> wrote:

> This looks like a bug to me. Jython cuts out jython.jar location from
> classpath and appends Lib to it. But, in general on TT jython,jar is not
> available and its "merged" into job.jar by pig. Hence, imports will always
> fail.
>
> ~Aniket
>
>
> On Mon, Mar 12, 2012 at 12:54 AM, Aniket Mokashi <[EMAIL PROTECTED]>wrote:
>
>> I spent some time debugging this. The reason is --
>>
>> Sys.path on TT for jython is - ['__classpath__', '__pyclasspath__/']
>>
>> And for client is ['', '/users/lib/Lib',
>> '/users/lib/jython_simplejson.jar/Lib', '__classpath__', '__pyclasspath__/']
>>
>> I am still figuring out why CLASSPATH (java.class.path property) on
>> tasktracker doesn't have job.jar on it. Hints anyone?
>>
>>
>> Thanks,
>>
>> Aniket
>>
>> On Tue, Oct 18, 2011 at 9:54 AM, Stan Rosenberg <
>> [EMAIL PROTECTED]> wrote:
>>
>>> Hi Clay,
>>>
>>> I am running a very recent version (one that contains this patch) of
>>> pig which was compiled from the trunk.
>>> How can I examine the jar file to determine which jython modules have
>>> been added?
>>>
>>> Thanks,
>>>
>>> stan
>>>
>>> On Tue, Oct 18, 2011 at 12:38 PM, Clay B. <[EMAIL PROTECTED]> wrote:
>>> > Hi Stan,
>>> >
>>> > I believe you are hitting
>>> https://issues.apache.org/jira/browse/PIG-1824
>>> >
>>> > -Clay
>>> >
>>> > On Mon, 17 Oct 2011, Stan Rosenberg wrote:
>>> >
>>> >> Hi,
>>> >>
>>> >> What's a proper way to deploy python udfs? I've dropped the latest
>>> >> version of jython.jar in $PIG_HOME/lib.
>>> >> Things work in "local" mode, but when I run on a cluster, built-in
>>> >> python modules cannot be found. E.g., urlparse cannot be located:
>>> >>
>>> >> ImportError: No module named urlparse
>>> >>
>>> >>       at
>>> org.python.core.PyException.fillInStackTrace(PyException.java:70)
>>> >>       at java.lang.Throwable.<init>(Throwable.java:181)
>>> >>       at java.lang.Exception.<init>(Exception.java:29)
>>> >>       at java.lang.RuntimeException.<init>(RuntimeException.java:32)
>>> >>       at org.python.core.PyException.<init>(PyException.java:46)
>>> >>       at org.python.core.PyException.<init>(PyException.java:43)
>>> >>       at org.python.core.PyException.<init>(PyException.java:61)
>>> >>       at org.python.core.Py.ImportError(Py.java:290)
>>> >>       at org.python.core.imp.import_first(imp.java:750)
>>> >>       at org.python.core.imp.import_name(imp.java:834)
>>> >>       at org.python.core.imp.importName(imp.java:884)
>>> >>       at
>>> org.python.core.ImportFunction.__call__(__builtin__.java:1220)
>>> >>       at org.python.core.PyObject.__call__(PyObject.java:357)
>>> >>       at org.python.core.__builtin__.__import__(__builtin__.java:1173)
>>> >>       at org.python.core.imp.importFromAs(imp.java:978)
>>> >>       at org.python.core.imp.importFrom(imp.java:954)
>>> >>       at org.python.pycode._pyx3.f$0(udfs.py:40)
>>> >>       at org.python.pycode._pyx3.call_function(udfs.py)
>>> >>       at org.python.core.PyTableCode.call(PyTableCode.java:165)
>>> >>       at org.python.core.PyCode.call(PyCode.java:18)
>>> >>       at org.python.core.Py.runCode(Py.java:1261)
>>> >>       at
>>> >> org.python.util.PythonInterpreter.execfile(PythonInterpreter.java:235)
>>> >>       at
>>> >>
>>> org.apache.pig.scripting.jython.JythonScriptEngine$Interpreter.execfile(JythonScriptEngine.java:176)
>>> >>       ... 15 more
>>> >>
>>> >> Thanks,
>>> >>
>>> >
"...:::Aniket:::... Quetzalco@tl"