Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Pig >> mail # user >> Need example of python code with dependency files


+
Ryan Compton 2013-11-05, 23:39
Copy link to this message
-
RE: Need example of python code with dependency files
You said "The .py code takes input from sys.stdin and outputs to sys.stdout" so I infer you are talking about streaming, not a python UDF. In that case, rather than streaming through your python script P.py, instead stream through a shell script S.sh. The shell script can untar shipped or cached files, then run P.py.  Maybe not completely elegant, but this approach works for me.
William F Dowling
Senior Technologist
Thomson Reuters

-----Original Message-----
From: Ryan Compton [mailto:[EMAIL PROTECTED]]
Sent: Tuesday, November 05, 2013 6:40 PM
To: [EMAIL PROTECTED]
Subject: Need example of python code with dependency files

I have some python code I'd like to deploy with a pig script. The .py
code takes input from sys.stdin and outputs to sys.stdout. It also
needs some parameter files to run properly.

The book "Programming Pig" tells me:

"The workaround for this is to create a TAR file and ship that, and
then have a step in your executable that unbundles the TAR file"

but I'm still real confused. I need an example of how to unpack it (eg
unpack in a bash script I pass around, unpack in the pig script, ??)
+
Ryan Compton 2013-11-09, 20:48