Hi all,

Just want to share with you my experience on using hdfs's glob syntax in
shell, it takes me a long time to figure out the problem.

I have a program that will accept an input path-pattern, and I will run
command like this "hadoop jar MainClass -input /2011/08/{01,02}"
And if you run this command, shell will expand this command to  "hadoop jar
MainClass -input /2011/08/01  /2011/08/02", then you will find that you will
always load the data of /2011/08/01 and the data of /2011/08/02 won't be
loaded
One solution to fix this problem is putting quotation around the
path-pattern to prevent shell syntax expansion, invoke command like this
 "hadoop jar MainClass -input '/2011/08/{01,02}' "

--
Best Regards

Jeff Zhang

Jeff,

Yes, this is a shell feature where it would glob anything that's
globby looking to it. Quotations is one way, but I prefer:

alias hadoop='noglob hadoop' which would prevent globbing in my zsh
for the hadoop executable.

On Thu, Aug 11, 2011 at 5:57 AM, Jeff Zhang <[EMAIL PROTECTED]> wrote:
> Hi all,
> Just want to share with you my experience on using hdfs's glob syntax in
> shell, it takes me a long time to figure out the problem.
> I have a program that will accept an input path-pattern, and I will run
> command like this "hadoop jar MainClass -input /2011/08/{01,02}"
> And if you run this command, shell will expand this command to  "hadoop jar
> MainClass -input /2011/08/01  /2011/08/02", then you will find that you will
> always load the data of /2011/08/01 and the data of /2011/08/02 won't be
> loaded
> One solution to fix this problem is putting quotation around the
> path-pattern to prevent shell syntax expansion, invoke command like this
>  "hadoop jar MainClass -input '/2011/08/{01,02}' "
> --
> Best Regards
>
> Jeff Zhang
>

--
Harsh J