Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Pig >> mail # user >> Need example of python code with dependency files


+
Ryan Compton 2013-11-05, 23:39
Copy link to this message
-
RE: Need example of python code with dependency files
You said "The .py code takes input from sys.stdin and outputs to sys.stdout" so I infer you are talking about streaming, not a python UDF. In that case, rather than streaming through your python script P.py, instead stream through a shell script S.sh. The shell script can untar shipped or cached files, then run P.py.  Maybe not completely elegant, but this approach works for me.
William F Dowling
Senior Technologist
Thomson Reuters

-----Original Message-----
From: Ryan Compton [mailto:[EMAIL PROTECTED]]
Sent: Tuesday, November 05, 2013 6:40 PM
To: [EMAIL PROTECTED]
Subject: Need example of python code with dependency files

I have some python code I'd like to deploy with a pig script. The .py
code takes input from sys.stdin and outputs to sys.stdout. It also
needs some parameter files to run properly.

The book "Programming Pig" tells me:

"The workaround for this is to create a TAR file and ship that, and
then have a step in your executable that unbundles the TAR file"

but I'm still real confused. I need an example of how to unpack it (eg
unpack in a bash script I pass around, unpack in the pig script, ??)
+
Ryan Compton 2013-11-09, 20:48
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB