Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig, mail # user - Need example of python code with dependency files


Copy link to this message
-
Re: Need example of python code with dependency files
Ryan Compton 2013-11-09, 20:48
Thanks, this worked for some test programs, but then I ran into some
other problems when I had to import things (for some reason pig was
looking for python2.4 stuff even though I explicitly told it 2.7, I'm
assuming this is more of a CentOS/Jython issue).

I ended up using PyInstaller to package up my code and streammed though that.

On Wed, Nov 6, 2013 at 6:23 AM,  <[EMAIL PROTECTED]> wrote:
> You said "The .py code takes input from sys.stdin and outputs to sys.stdout" so I infer you are talking about streaming, not a python UDF. In that case, rather than streaming through your python script P.py, instead stream through a shell script S.sh. The shell script can untar shipped or cached files, then run P.py.  Maybe not completely elegant, but this approach works for me.
>
>
> William F Dowling
> Senior Technologist
> Thomson Reuters
>
> -----Original Message-----
> From: Ryan Compton [mailto:[EMAIL PROTECTED]]
> Sent: Tuesday, November 05, 2013 6:40 PM
> To: [EMAIL PROTECTED]
> Subject: Need example of python code with dependency files
>
> I have some python code I'd like to deploy with a pig script. The .py
> code takes input from sys.stdin and outputs to sys.stdout. It also
> needs some parameter files to run properly.
>
> The book "Programming Pig" tells me:
>
> "The workaround for this is to create a TAR file and ship that, and
> then have a step in your executable that unbundles the TAR file"
>
> but I'm still real confused. I need an example of how to unpack it (eg
> unpack in a bash script I pass around, unpack in the pig script, ??)