Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
HDFS >> mail # user >> HDFS's glob syntax conflict with Shell's syntax expansion


Copy link to this message
-
HDFS's glob syntax conflict with Shell's syntax expansion
Hi all,

Just want to share with you my experience on using hdfs's glob syntax in
shell, it takes me a long time to figure out the problem.

I have a program that will accept an input path-pattern, and I will run
command like this "hadoop jar MainClass -input /2011/08/{01,02}"
And if you run this command, shell will expand this command to  "hadoop jar
MainClass -input /2011/08/01  /2011/08/02", then you will find that you will
always load the data of /2011/08/01 and the data of /2011/08/02 won't be
loaded
One solution to fix this problem is putting quotation around the
path-pattern to prevent shell syntax expansion, invoke command like this
 "hadoop jar MainClass -input '/2011/08/{01,02}' "

--
Best Regards

Jeff Zhang
+
Harsh J 2011-08-11, 05:30