Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Pig >> mail # user >> python modules


+
Stan Rosenberg 2011-10-18, 02:49
+
Clay B. 2011-10-18, 16:38
+
Stan Rosenberg 2011-10-18, 16:54
+
Aniket Mokashi 2012-03-12, 07:54
Copy link to this message
-
Re: python modules
This looks like a bug to me. Jython cuts out jython.jar location from
classpath and appends Lib to it. But, in general on TT jython,jar is not
available and its "merged" into job.jar by pig. Hence, imports will always
fail.

~Aniket

On Mon, Mar 12, 2012 at 12:54 AM, Aniket Mokashi <[EMAIL PROTECTED]>wrote:

> I spent some time debugging this. The reason is --
>
> Sys.path on TT for jython is - ['__classpath__', '__pyclasspath__/']
>
> And for client is ['', '/users/lib/Lib',
> '/users/lib/jython_simplejson.jar/Lib', '__classpath__', '__pyclasspath__/']
>
> I am still figuring out why CLASSPATH (java.class.path property) on
> tasktracker doesn't have job.jar on it. Hints anyone?
>
>
> Thanks,
>
> Aniket
>
> On Tue, Oct 18, 2011 at 9:54 AM, Stan Rosenberg <
> [EMAIL PROTECTED]> wrote:
>
>> Hi Clay,
>>
>> I am running a very recent version (one that contains this patch) of
>> pig which was compiled from the trunk.
>> How can I examine the jar file to determine which jython modules have
>> been added?
>>
>> Thanks,
>>
>> stan
>>
>> On Tue, Oct 18, 2011 at 12:38 PM, Clay B. <[EMAIL PROTECTED]> wrote:
>> > Hi Stan,
>> >
>> > I believe you are hitting
>> https://issues.apache.org/jira/browse/PIG-1824
>> >
>> > -Clay
>> >
>> > On Mon, 17 Oct 2011, Stan Rosenberg wrote:
>> >
>> >> Hi,
>> >>
>> >> What's a proper way to deploy python udfs? I've dropped the latest
>> >> version of jython.jar in $PIG_HOME/lib.
>> >> Things work in "local" mode, but when I run on a cluster, built-in
>> >> python modules cannot be found. E.g., urlparse cannot be located:
>> >>
>> >> ImportError: No module named urlparse
>> >>
>> >>       at
>> org.python.core.PyException.fillInStackTrace(PyException.java:70)
>> >>       at java.lang.Throwable.<init>(Throwable.java:181)
>> >>       at java.lang.Exception.<init>(Exception.java:29)
>> >>       at java.lang.RuntimeException.<init>(RuntimeException.java:32)
>> >>       at org.python.core.PyException.<init>(PyException.java:46)
>> >>       at org.python.core.PyException.<init>(PyException.java:43)
>> >>       at org.python.core.PyException.<init>(PyException.java:61)
>> >>       at org.python.core.Py.ImportError(Py.java:290)
>> >>       at org.python.core.imp.import_first(imp.java:750)
>> >>       at org.python.core.imp.import_name(imp.java:834)
>> >>       at org.python.core.imp.importName(imp.java:884)
>> >>       at org.python.core.ImportFunction.__call__(__builtin__.java:1220)
>> >>       at org.python.core.PyObject.__call__(PyObject.java:357)
>> >>       at org.python.core.__builtin__.__import__(__builtin__.java:1173)
>> >>       at org.python.core.imp.importFromAs(imp.java:978)
>> >>       at org.python.core.imp.importFrom(imp.java:954)
>> >>       at org.python.pycode._pyx3.f$0(udfs.py:40)
>> >>       at org.python.pycode._pyx3.call_function(udfs.py)
>> >>       at org.python.core.PyTableCode.call(PyTableCode.java:165)
>> >>       at org.python.core.PyCode.call(PyCode.java:18)
>> >>       at org.python.core.Py.runCode(Py.java:1261)
>> >>       at
>> >> org.python.util.PythonInterpreter.execfile(PythonInterpreter.java:235)
>> >>       at
>> >>
>> org.apache.pig.scripting.jython.JythonScriptEngine$Interpreter.execfile(JythonScriptEngine.java:176)
>> >>       ... 15 more
>> >>
>> >> Thanks,
>> >>
>> >> stan
>> >>
>> >
>>
>
>
>
> --
> "...:::Aniket:::... Quetzalco@tl"
>

--
"...:::Aniket:::... Quetzalco@tl"
+
Aniket Mokashi 2012-03-14, 06:31
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB