Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig >> mail # user >> Import libraries in Jython UDFs


Copy link to this message
-
Re: Import libraries in Jython UDFs
Hi Russell,

Check if your hadoop has MAPREDUCE-967. If not, pig's job.jar will be
unjared on the tasks. With that, jython's static method's do not put /Lib
in python sys path. You can use the instrumentation code i shared in your
python udf file and find the sys.path on task nodes to confirm this. If
/Lib is on sys.path, all imports will work.

One workaround is to put jython jar on all nodes of the cluster under
hadoop lib.

~Aniket

On Tue, Jul 24, 2012 at 5:21 PM, Russell Jurney <[EMAIL PROTECTED]>wrote:

> Thanks, but that isn't my issue. I am unable to import any packages.
> Trying to get the path right...
>
> Russell Jurney http://datasyndrome.com
>
> On Jul 24, 2012, at 10:00 AM, Chun Yang <[EMAIL PROTECTED]>
> wrote:
>
> > Hi Russell,
> >
> > Are you able to import other modules beside email? If not, maybe this is
> > related to your problem: https://issues.apache.org/jira/browse/PIG-2665
> >
> > -Chun
> >
> > On 7/23/12 11:26 PM, "Russell Jurney" <[EMAIL PROTECTED]> wrote:
> >
> >> ls /me/jython2.5.2/Lib/
> >>
> >> tons of class files...
> >> email/
> >>
> >>
> >> This is in local mode, atm. I add this directory to my java classpath,
> >> check.
> >>
> >> On Mon, Jul 23, 2012 at 11:10 PM, Aniket Mokashi <[EMAIL PROTECTED]
> >wrote:
> >>
> >>> jar tf jython.jar | grep email
> >>>
> >>> Having jar in PIG_CLASSPATH would work if you have
> >>> https://issues.apache.org/jira/browse/MAPREDUCE-967.
> >>>
> >>> You can use following to debug the sys.path on tasknodes-
> >>>
> >>> from java.lang import System
> >>> print "python.home "
> >>> print System.getProperties().getProperty("python.home")
> >>> print "java.class.path "
> >>> print System.getProperties().getProperty("java.class.path")
> >>> print "install.root "
> >>> print System.getProperties().getProperty("install.root")
> >>> print "python.home "
> >>> print System.getProperties().getProperty("python.home")
> >>>
> >>> ~Aniket
> >>>
> >>> On Mon, Jul 23, 2012 at 6:33 PM, Russell Jurney <
> [EMAIL PROTECTED]
> >>>> wrote:
> >>>
> >>>> No, how do I find which jar the email package is in?
> >>>>
> >>>> On Mon, Jul 23, 2012 at 6:02 PM, Norbert Burger <
> >>> [EMAIL PROTECTED]
> >>>>> wrote:
> >>>>
> >>>>> Have you registered the JAR in your Pig script (for local mode) and
> >>>>> also added it to PIG_CLASSPATH (for remote mode, to get it into the
> >>>>> distributed cache)?
> >>>>>
> >>>>> Norbert
> >>>>>
> >>>>> On Mon, Jul 23, 2012 at 8:33 PM, Russell Jurney
> >>>>> <[EMAIL PROTECTED]> wrote:
> >>>>>> The email package is a part of Jython, I believe:
> >>>>>> http://www.jython.org/docs/library/email.html
> >>>>>>
> >>>>>> However, when I 'import email' in udfs.py, I get this error:
> >>>>>>
> >>>>>> 2012-07-23 17:32:51,027 [main] ERROR
> >>> org.apache.pig.tools.grunt.Grunt -
> >>>>>> ERROR 1121: Python Error. Traceback (most recent call last):
> >>>>>>  File "/Users/rjurney/Collecting-Data/src/pig/udfs.py", line 1, in
> >>>>> <module>
> >>>>>>    import email
> >>>>>> ImportError: No module named email
> >>>>>>
> >>>>>>
> >>>>>> How do I import and use built-in packages in Jython?
> >>>>>>
> >>>>>> --
> >>>>>> Russell Jurney twitter.com/rjurney [EMAIL PROTECTED]
> >>>>> datasyndrome.com
> >>>>>
> >>>>
> >>>>
> >>>>
> >>>> --
> >>>> Russell Jurney twitter.com/rjurney [EMAIL PROTECTED]
> >>>> datasyndrome.com
> >>>>
> >>>
> >>>
> >>>
> >>> --
> >>> "...:::Aniket:::... Quetzalco@tl"
> >>>
> >>
> >>
> >>
> >> --
> >> Russell Jurney twitter.com/rjurney [EMAIL PROTECTED]
> datasyndrome.com
> >
>

--
"...:::Aniket:::... Quetzalco@tl"