Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig >> mail # user >> The fact that STORE needs to use quoted file names but fs does not is confusing!


Copy link to this message
-
Re: The fact that STORE needs to use quoted file names but fs does not is confusing!
Also, 'local' is overloaded here. local doesn't mean local fs, local means
local map-reduce mode (non-cluster)

Ashutosh
On Wed, Aug 24, 2011 at 14:31, Dmitriy Ryaboy <[EMAIL PROTECTED]> wrote:

> fs delegates to "hadoop fs" (so, effectively, you are running "hadoop fs
> -ls", and behaves exactly the same way as the hadoop command.
>
> Not sure you can get hadoop fs commands to work properly with local
> filesystem.
>
> D
>
>
>
> On Wed, Aug 24, 2011 at 1:28 PM, Kevin Burton <[EMAIL PROTECTED]> wrote:
>
> > This just bit me.
> >
> > I can do:
> >
> > STORE data INTO '/tmp/brokenfs.out';
> >
> > but
> >
> > fs -ls '/tmp/brokenfs.out';
> >
> > won't work because it can't be quoted.
> >
> > fs -ls /tmp/brokenfs.out;
> >
> > works though.
> >
> >
> > ………...
> >
> > I'm pretty sure this is a bug.
> >
> > fs won't ls or rm any files on the local file system when run with -x
> local
> > mode.
> >
> > java -Xmx384M -classpath
> > pig/pig-0.9.0.jar:pig/lib/zebra.jar:../hadoop-0.20.2/conf/
> > org.apache.pig.Main -x local brokenfs.pig
> > …
> > ls: Cannot access '/tmp/brokenfs.out': No such file or directory.
> > 2011-08-24 13:20:55,405 [main] ERROR org.apache.pig.tools.grunt.Grunt -
> > ERROR 2997: Encountered IOException. fs command '-ls '/tmp/brokenfs.out''
> > failed. Please check output logs for details
> > Details at logfile: /Users/burton/projects/rankst3r/pig_1314217243730.log
> >
> > but then I run:
> >
> > > ls -al /tmp/brokenfs.out/
> > total 16
> > drwxr-xr-x   4 burton  wheel  136 Aug 24 13:25 ./
> > drwxrwxrwt  21 root    wheel  714 Aug 24 13:25 ../
> > -rw-r--r--   1 burton  wheel   12 Aug 24 13:25 .part-m-00000.crc
> > -rwxrwxrwx   1 burton  wheel    4 Aug 24 13:25 part-m-00000*
> >
> >
> > -- brokenfs.pig
> >
> >
> > data = LOAD 'brokenfs.csv' USING PigStorage(',') AS (foo:int, bar:int);
> >
> > STORE data INTO '/tmp/brokenfs.out';
> > exec;
> >
> > fs -ls '/tmp/brokenfs.out';
> >
> > -- brokenfs.csv
> >
> > 1,2
> >
> >
> > --
> >
> > Founder/CEO Spinn3r.com
> >
> > Location: *San Francisco, CA*
> > Skype: *burtonator*
> >
> > Skype-in: *(415) 871-0687*
> >
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB