Danfeng Li 2012-12-20, 18:01
Alan Gates 2012-12-20, 18:57
Danfeng Li 2012-12-20, 20:00
Thomas Bach 2012-12-21, 10:25
-Re: pig ship tar files
Rohini Palaniswamy 2012-12-27, 17:40
You can also use -Dmapred.cache.archives=<hdfs:///your tar file path> to
ship the tar file using distributed cache. Hadoop will take care of
untarring the file and putting it in the current directory if the extension
is one of .zip, .tar, .tgz or .tar.gz. This is a feature of
hadoop's distributed cache.
On Fri, Dec 21, 2012 at 2:25 AM, Thomas Bach
> On Thu, Dec 20, 2012 at 01:01:49PM -0500, Danfeng Li wrote:
> > I read alot of about pig can ship a tar file and untar it before
> > execution. However, I couldn't find any example. Can someone provide
> > an example?
> The trick is to use the `SH' statement to untar the file.
> > What I would like to do is to ship a python module, such as nltk,
> > for my streaming.
> Try something like (untested)
> DEFINE my_cmd `relative/path/to/my_cmd/in/tar/file.py`
> SH tar xf nltk.tar
> Does this help/work?