Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig >> mail # user >> Need example of python code with dependency files


Copy link to this message
-
Re: Need example of python code with dependency files
Thanks, this worked for some test programs, but then I ran into some
other problems when I had to import things (for some reason pig was
looking for python2.4 stuff even though I explicitly told it 2.7, I'm
assuming this is more of a CentOS/Jython issue).

I ended up using PyInstaller to package up my code and streammed though that.

On Wed, Nov 6, 2013 at 6:23 AM,  <[EMAIL PROTECTED]> wrote:
> You said "The .py code takes input from sys.stdin and outputs to sys.stdout" so I infer you are talking about streaming, not a python UDF. In that case, rather than streaming through your python script P.py, instead stream through a shell script S.sh. The shell script can untar shipped or cached files, then run P.py.  Maybe not completely elegant, but this approach works for me.
>
>
> William F Dowling
> Senior Technologist
> Thomson Reuters
>
> -----Original Message-----
> From: Ryan Compton [mailto:[EMAIL PROTECTED]]
> Sent: Tuesday, November 05, 2013 6:40 PM
> To: [EMAIL PROTECTED]
> Subject: Need example of python code with dependency files
>
> I have some python code I'd like to deploy with a pig script. The .py
> code takes input from sys.stdin and outputs to sys.stdout. It also
> needs some parameter files to run properly.
>
> The book "Programming Pig" tells me:
>
> "The workaround for this is to create a TAR file and ship that, and
> then have a step in your executable that unbundles the TAR file"
>
> but I'm still real confused. I need an example of how to unpack it (eg
> unpack in a bash script I pass around, unpack in the pig script, ??)
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB