Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Pig >> mail # user >> Import libraries in Jython UDFs


+
Russell Jurney 2012-07-24, 00:33
+
Norbert Burger 2012-07-24, 01:02
+
Russell Jurney 2012-07-24, 01:33
+
Aniket Mokashi 2012-07-24, 06:10
+
Russell Jurney 2012-07-24, 06:26
+
Chun Yang 2012-07-24, 17:00
+
Russell Jurney 2012-07-25, 00:21
+
Aniket Mokashi 2012-07-26, 00:10
+
Russell Jurney 2012-07-26, 05:16
Copy link to this message
-
Re: Import libraries in Jython UDFs
Fixed by https://issues.apache.org/jira/browse/PIG-2665 in Pig 0.11
On Wed, Jul 25, 2012 at 10:16 PM, Russell Jurney
<[EMAIL PROTECTED]>wrote:

> Does Jython.jar have the base libs?
>
> I'm running this in local mode, btw.
>
> Russell Jurney
> twitter.com/rjurney
> [EMAIL PROTECTED]
> datasyndrome.com
>
> On Jul 25, 2012, at 5:11 PM, Aniket Mokashi <[EMAIL PROTECTED]> wrote:
>
> > Hi Russell,
> >
> > Check if your hadoop has MAPREDUCE-967. If not, pig's job.jar will be
> > unjared on the tasks. With that, jython's static method's do not put /Lib
> > in python sys path. You can use the instrumentation code i shared in your
> > python udf file and find the sys.path on task nodes to confirm this. If
> > /Lib is on sys.path, all imports will work.
> >
> > One workaround is to put jython jar on all nodes of the cluster under
> > hadoop lib.
> >
> > ~Aniket
> >
> > On Tue, Jul 24, 2012 at 5:21 PM, Russell Jurney <
> [EMAIL PROTECTED]>wrote:
> >
> >> Thanks, but that isn't my issue. I am unable to import any packages.
> >> Trying to get the path right...
> >>
> >> Russell Jurney http://datasyndrome.com
> >>
> >> On Jul 24, 2012, at 10:00 AM, Chun Yang <
> [EMAIL PROTECTED]>
> >> wrote:
> >>
> >>> Hi Russell,
> >>>
> >>> Are you able to import other modules beside email? If not, maybe this
> is
> >>> related to your problem:
> https://issues.apache.org/jira/browse/PIG-2665
> >>>
> >>> -Chun
> >>>
> >>> On 7/23/12 11:26 PM, "Russell Jurney" <[EMAIL PROTECTED]>
> wrote:
> >>>
> >>>> ls /me/jython2.5.2/Lib/
> >>>>
> >>>> tons of class files...
> >>>> email/
> >>>>
> >>>>
> >>>> This is in local mode, atm. I add this directory to my java classpath,
> >>>> check.
> >>>>
> >>>> On Mon, Jul 23, 2012 at 11:10 PM, Aniket Mokashi <[EMAIL PROTECTED]
> >>> wrote:
> >>>>
> >>>>> jar tf jython.jar | grep email
> >>>>>
> >>>>> Having jar in PIG_CLASSPATH would work if you have
> >>>>> https://issues.apache.org/jira/browse/MAPREDUCE-967.
> >>>>>
> >>>>> You can use following to debug the sys.path on tasknodes-
> >>>>>
> >>>>> from java.lang import System
> >>>>> print "python.home "
> >>>>> print System.getProperties().getProperty("python.home")
> >>>>> print "java.class.path "
> >>>>> print System.getProperties().getProperty("java.class.path")
> >>>>> print "install.root "
> >>>>> print System.getProperties().getProperty("install.root")
> >>>>> print "python.home "
> >>>>> print System.getProperties().getProperty("python.home")
> >>>>>
> >>>>> ~Aniket
> >>>>>
> >>>>> On Mon, Jul 23, 2012 at 6:33 PM, Russell Jurney <
> >> [EMAIL PROTECTED]
> >>>>>> wrote:
> >>>>>
> >>>>>> No, how do I find which jar the email package is in?
> >>>>>>
> >>>>>> On Mon, Jul 23, 2012 at 6:02 PM, Norbert Burger <
> >>>>> [EMAIL PROTECTED]
> >>>>>>> wrote:
> >>>>>>
> >>>>>>> Have you registered the JAR in your Pig script (for local mode) and
> >>>>>>> also added it to PIG_CLASSPATH (for remote mode, to get it into the
> >>>>>>> distributed cache)?
> >>>>>>>
> >>>>>>> Norbert
> >>>>>>>
> >>>>>>> On Mon, Jul 23, 2012 at 8:33 PM, Russell Jurney
> >>>>>>> <[EMAIL PROTECTED]> wrote:
> >>>>>>>> The email package is a part of Jython, I believe:
> >>>>>>>> http://www.jython.org/docs/library/email.html
> >>>>>>>>
> >>>>>>>> However, when I 'import email' in udfs.py, I get this error:
> >>>>>>>>
> >>>>>>>> 2012-07-23 17:32:51,027 [main] ERROR
> >>>>> org.apache.pig.tools.grunt.Grunt -
> >>>>>>>> ERROR 1121: Python Error. Traceback (most recent call last):
> >>>>>>>> File "/Users/rjurney/Collecting-Data/src/pig/udfs.py", line 1, in
> >>>>>>> <module>
> >>>>>>>>   import email
> >>>>>>>> ImportError: No module named email
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> How do I import and use built-in packages in Jython?
> >>>>>>>>
> >>>>>>>> --
> >>>>>>>> Russell Jurney twitter.com/rjurney [EMAIL PROTECTED]
> >>>>>>> datasyndrome.com
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> --
> >>>>>> Russell Jurney twitter.com/rjurney [EMAIL PROTECTED]

Russell Jurney twitter.com/rjurney [EMAIL PROTECTED] datasyndrome.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB