Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig, mail # user - The fact that STORE needs to use quoted file names but fs does not is confusing!


Copy link to this message
-
Re: The fact that STORE needs to use quoted file names but fs does not is confusing!
Ashutosh Chauhan 2011-08-24, 21:37
Also, 'local' is overloaded here. local doesn't mean local fs, local means
local map-reduce mode (non-cluster)

Ashutosh
On Wed, Aug 24, 2011 at 14:31, Dmitriy Ryaboy <[EMAIL PROTECTED]> wrote:

> fs delegates to "hadoop fs" (so, effectively, you are running "hadoop fs
> -ls", and behaves exactly the same way as the hadoop command.
>
> Not sure you can get hadoop fs commands to work properly with local
> filesystem.
>
> D
>
>
>
> On Wed, Aug 24, 2011 at 1:28 PM, Kevin Burton <[EMAIL PROTECTED]> wrote:
>
> > This just bit me.
> >
> > I can do:
> >
> > STORE data INTO '/tmp/brokenfs.out';
> >
> > but
> >
> > fs -ls '/tmp/brokenfs.out';
> >
> > won't work because it can't be quoted.
> >
> > fs -ls /tmp/brokenfs.out;
> >
> > works though.
> >
> >
> > ………...
> >
> > I'm pretty sure this is a bug.
> >
> > fs won't ls or rm any files on the local file system when run with -x
> local
> > mode.
> >
> > java -Xmx384M -classpath
> > pig/pig-0.9.0.jar:pig/lib/zebra.jar:../hadoop-0.20.2/conf/
> > org.apache.pig.Main -x local brokenfs.pig
> > …
> > ls: Cannot access '/tmp/brokenfs.out': No such file or directory.
> > 2011-08-24 13:20:55,405 [main] ERROR org.apache.pig.tools.grunt.Grunt -
> > ERROR 2997: Encountered IOException. fs command '-ls '/tmp/brokenfs.out''
> > failed. Please check output logs for details
> > Details at logfile: /Users/burton/projects/rankst3r/pig_1314217243730.log
> >
> > but then I run:
> >
> > > ls -al /tmp/brokenfs.out/
> > total 16
> > drwxr-xr-x   4 burton  wheel  136 Aug 24 13:25 ./
> > drwxrwxrwt  21 root    wheel  714 Aug 24 13:25 ../
> > -rw-r--r--   1 burton  wheel   12 Aug 24 13:25 .part-m-00000.crc
> > -rwxrwxrwx   1 burton  wheel    4 Aug 24 13:25 part-m-00000*
> >
> >
> > -- brokenfs.pig
> >
> >
> > data = LOAD 'brokenfs.csv' USING PigStorage(',') AS (foo:int, bar:int);
> >
> > STORE data INTO '/tmp/brokenfs.out';
> > exec;
> >
> > fs -ls '/tmp/brokenfs.out';
> >
> > -- brokenfs.csv
> >
> > 1,2
> >
> >
> > --
> >
> > Founder/CEO Spinn3r.com
> >
> > Location: *San Francisco, CA*
> > Skype: *burtonator*
> >
> > Skype-in: *(415) 871-0687*
> >
>