Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig, mail # user - Import libraries in Jython UDFs


Copy link to this message
-
Re: Import libraries in Jython UDFs
Russell Jurney 2012-07-26, 05:16
Does Jython.jar have the base libs?

I'm running this in local mode, btw.

Russell Jurney
twitter.com/rjurney
[EMAIL PROTECTED]
datasyndrome.com

On Jul 25, 2012, at 5:11 PM, Aniket Mokashi <[EMAIL PROTECTED]> wrote:

> Hi Russell,
>
> Check if your hadoop has MAPREDUCE-967. If not, pig's job.jar will be
> unjared on the tasks. With that, jython's static method's do not put /Lib
> in python sys path. You can use the instrumentation code i shared in your
> python udf file and find the sys.path on task nodes to confirm this. If
> /Lib is on sys.path, all imports will work.
>
> One workaround is to put jython jar on all nodes of the cluster under
> hadoop lib.
>
> ~Aniket
>
> On Tue, Jul 24, 2012 at 5:21 PM, Russell Jurney <[EMAIL PROTECTED]>wrote:
>
>> Thanks, but that isn't my issue. I am unable to import any packages.
>> Trying to get the path right...
>>
>> Russell Jurney http://datasyndrome.com
>>
>> On Jul 24, 2012, at 10:00 AM, Chun Yang <[EMAIL PROTECTED]>
>> wrote:
>>
>>> Hi Russell,
>>>
>>> Are you able to import other modules beside email? If not, maybe this is
>>> related to your problem: https://issues.apache.org/jira/browse/PIG-2665
>>>
>>> -Chun
>>>
>>> On 7/23/12 11:26 PM, "Russell Jurney" <[EMAIL PROTECTED]> wrote:
>>>
>>>> ls /me/jython2.5.2/Lib/
>>>>
>>>> tons of class files...
>>>> email/
>>>>
>>>>
>>>> This is in local mode, atm. I add this directory to my java classpath,
>>>> check.
>>>>
>>>> On Mon, Jul 23, 2012 at 11:10 PM, Aniket Mokashi <[EMAIL PROTECTED]
>>> wrote:
>>>>
>>>>> jar tf jython.jar | grep email
>>>>>
>>>>> Having jar in PIG_CLASSPATH would work if you have
>>>>> https://issues.apache.org/jira/browse/MAPREDUCE-967.
>>>>>
>>>>> You can use following to debug the sys.path on tasknodes-
>>>>>
>>>>> from java.lang import System
>>>>> print "python.home "
>>>>> print System.getProperties().getProperty("python.home")
>>>>> print "java.class.path "
>>>>> print System.getProperties().getProperty("java.class.path")
>>>>> print "install.root "
>>>>> print System.getProperties().getProperty("install.root")
>>>>> print "python.home "
>>>>> print System.getProperties().getProperty("python.home")
>>>>>
>>>>> ~Aniket
>>>>>
>>>>> On Mon, Jul 23, 2012 at 6:33 PM, Russell Jurney <
>> [EMAIL PROTECTED]
>>>>>> wrote:
>>>>>
>>>>>> No, how do I find which jar the email package is in?
>>>>>>
>>>>>> On Mon, Jul 23, 2012 at 6:02 PM, Norbert Burger <
>>>>> [EMAIL PROTECTED]
>>>>>>> wrote:
>>>>>>
>>>>>>> Have you registered the JAR in your Pig script (for local mode) and
>>>>>>> also added it to PIG_CLASSPATH (for remote mode, to get it into the
>>>>>>> distributed cache)?
>>>>>>>
>>>>>>> Norbert
>>>>>>>
>>>>>>> On Mon, Jul 23, 2012 at 8:33 PM, Russell Jurney
>>>>>>> <[EMAIL PROTECTED]> wrote:
>>>>>>>> The email package is a part of Jython, I believe:
>>>>>>>> http://www.jython.org/docs/library/email.html
>>>>>>>>
>>>>>>>> However, when I 'import email' in udfs.py, I get this error:
>>>>>>>>
>>>>>>>> 2012-07-23 17:32:51,027 [main] ERROR
>>>>> org.apache.pig.tools.grunt.Grunt -
>>>>>>>> ERROR 1121: Python Error. Traceback (most recent call last):
>>>>>>>> File "/Users/rjurney/Collecting-Data/src/pig/udfs.py", line 1, in
>>>>>>> <module>
>>>>>>>>   import email
>>>>>>>> ImportError: No module named email
>>>>>>>>
>>>>>>>>
>>>>>>>> How do I import and use built-in packages in Jython?
>>>>>>>>
>>>>>>>> --
>>>>>>>> Russell Jurney twitter.com/rjurney [EMAIL PROTECTED]
>>>>>>> datasyndrome.com
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Russell Jurney twitter.com/rjurney [EMAIL PROTECTED]
>>>>>> datasyndrome.com
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> "...:::Aniket:::... Quetzalco@tl"
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Russell Jurney twitter.com/rjurney [EMAIL PROTECTED]
>> datasyndrome.com
>>>
>>
>
>
>
> --
> "...:::Aniket:::... Quetzalco@tl"