Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
HDFS >> mail # user >> HDFS's glob syntax conflict with Shell's syntax expansion


Copy link to this message
-
HDFS's glob syntax conflict with Shell's syntax expansion
Hi all,

Just want to share with you my experience on using hdfs's glob syntax in
shell, it takes me a long time to figure out the problem.

I have a program that will accept an input path-pattern, and I will run
command like this "hadoop jar MainClass -input /2011/08/{01,02}"
And if you run this command, shell will expand this command to  "hadoop jar
MainClass -input /2011/08/01  /2011/08/02", then you will find that you will
always load the data of /2011/08/01 and the data of /2011/08/02 won't be
loaded
One solution to fix this problem is putting quotation around the
path-pattern to prevent shell syntax expansion, invoke command like this
 "hadoop jar MainClass -input '/2011/08/{01,02}' "

--
Best Regards

Jeff Zhang
+
Harsh J 2011-08-11, 05:30
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB