Hi guys,

      Just wanted to know how we can use "--query" attribute in
<command></command> of sqoop action in Oozie. The issue is that if we have
something like

* <command>blah blah ....  --query "select * from table"</command> *is
converted to

<arg>blah</arg>
<arg>blah</arg>
<arg>--query</arg>
<arg>"select</arg>
<arg>*</arg>
<arg>from</arg>
....so on
while what we want is

<arg>blah</arg>
<arg>blah</arg>
<arg>--query</arg>
<arg>"select * from table:</arg>
Is there a workaround?

Regards,
Nitin Kak

Hi Nikin,
workaround to use --query parameter inside <command> tag in Oozie is to use multiple <arg> tags instead :-)

Basically Oozie will blindly divide value supplied in <command> tag by spaces without taking into account any escaping. You are required to use <arg> tags instead of <command> in case that you need to specify arguments with spaces. For more information please consider Oozie Sqoop action documentation [1].

Jarcec

Links:
1: http://oozie.apache.org/docs/3.2.0-incubating/DG_SqoopActionExtension.html

On Fri, Oct 19, 2012 at 05:10:07PM -0400, Nitin kak wrote:
> Hi guys,
>
>       Just wanted to know how we can use "--query" attribute in
> <command></command> of sqoop action in Oozie. The issue is that if we have
> something like
>
> * <command>blah blah ....  --query "select * from table"</command> *is
> converted to
>
> <arg>blah</arg>
> <arg>blah</arg>
> <arg>--query</arg>
> <arg>"select</arg>
> <arg>*</arg>
> <arg>from</arg>
> ....so on
> while what we want is
>
> <arg>blah</arg>
> <arg>blah</arg>
> <arg>--query</arg>
> <arg>"select * from table:</arg>
>
>
> Is there a workaround?
>
> Regards,
> Nitin Kak

Ahh...that sucks..I hope they will come up with something in future
releases...

One follow up question...We can't use <command></command> in combination
with <arg></arg> right?

On Fri, Oct 19, 2012 at 5:25 PM, Jarek Jarcec Cecho <[EMAIL PROTECTED]>wrote:

> Hi Nikin,
> workaround to use --query parameter inside <command> tag in Oozie is to
> use multiple <arg> tags instead :-)
>
> Basically Oozie will blindly divide value supplied in <command> tag by
> spaces without taking into account any escaping. You are required to use
> <arg> tags instead of <command> in case that you need to specify arguments
> with spaces. For more information please consider Oozie Sqoop action
> documentation [1].
>
> Jarcec
>
> Links:
> 1:
> http://oozie.apache.org/docs/3.2.0-incubating/DG_SqoopActionExtension.html
>
> On Fri, Oct 19, 2012 at 05:10:07PM -0400, Nitin kak wrote:
> > Hi guys,
> >
> >       Just wanted to know how we can use "--query" attribute in
> > <command></command> of sqoop action in Oozie. The issue is that if we
> have
> > something like
> >
> > * <command>blah blah ....  --query "select * from table"</command> *is
> > converted to
> >
> > <arg>blah</arg>
> > <arg>blah</arg>
> > <arg>--query</arg>
> > <arg>"select</arg>
> > <arg>*</arg>
> > <arg>from</arg>
> > ....so on
> > while what we want is
> >
> > <arg>blah</arg>
> > <arg>blah</arg>
> > <arg>--query</arg>
> > <arg>"select * from table:</arg>
> >
> >
> > Is there a workaround?
> >
> > Regards,
> > Nitin Kak
>

That is correct - you can't use <command> tag and <arg> tags together. You can use either <command> or <arg>.

Jarcec

On Fri, Oct 19, 2012 at 05:32:12PM -0400, Nitin kak wrote:
> Ahh...that sucks..I hope they will come up with something in future
> releases...
>
> One follow up question...We can't use <command></command> in combination
> with <arg></arg> right?
>
> On Fri, Oct 19, 2012 at 5:25 PM, Jarek Jarcec Cecho <[EMAIL PROTECTED]>wrote:
>
> > Hi Nikin,
> > workaround to use --query parameter inside <command> tag in Oozie is to
> > use multiple <arg> tags instead :-)
> >
> > Basically Oozie will blindly divide value supplied in <command> tag by
> > spaces without taking into account any escaping. You are required to use
> > <arg> tags instead of <command> in case that you need to specify arguments
> > with spaces. For more information please consider Oozie Sqoop action
> > documentation [1].
> >
> > Jarcec
> >
> > Links:
> > 1:
> > http://oozie.apache.org/docs/3.2.0-incubating/DG_SqoopActionExtension.html
> >
> > On Fri, Oct 19, 2012 at 05:10:07PM -0400, Nitin kak wrote:
> > > Hi guys,
> > >
> > >       Just wanted to know how we can use "--query" attribute in
> > > <command></command> of sqoop action in Oozie. The issue is that if we
> > have
> > > something like
> > >
> > > * <command>blah blah ....  --query "select * from table"</command> *is
> > > converted to
> > >
> > > <arg>blah</arg>
> > > <arg>blah</arg>
> > > <arg>--query</arg>
> > > <arg>"select</arg>
> > > <arg>*</arg>
> > > <arg>from</arg>
> > > ....so on
> > > while what we want is
> > >
> > > <arg>blah</arg>
> > > <arg>blah</arg>
> > > <arg>--query</arg>
> > > <arg>"select * from table:</arg>
> > >
> > >
> > > Is there a workaround?
> > >
> > > Regards,
> > > Nitin Kak
> >

Hi Nitin,
it can actually get quite tricky to correctly split the command line when Oozie would allow all soft of different escaping inside <command> argument. That's why it fits for simple use cases and that's why there is tag <arg> for more advance use cases.

I can see why you need to generate your whole command line in your use case. I've actually done something similar (conditional sqoop command line generation) in the past, but I've used <arg> tags and just filled argument values (password, username, query). Is something like that possible in your use case?

Jarcec

On Mon, Oct 22, 2012 at 11:11:21AM -0400, Nitin kak wrote:
> Okay. So the thing is that my pre-compiler could have generated the whole
> command(which requires the --query "select a, b as c from .... " ) as a
> parameter. While what I now need is many more parameter values for
> --connect, --query, --username, --password and all.
>
> What I feel is that if sqoop command supports "--query" construct, then we
> should be able to use that construct in <command></command> in Oozie.
>
> Please let me know what you think.
>
> Regards,
> Nitin
>
> On Mon, Oct 22, 2012 at 11:01 AM, Jarek Jarcec Cecho <[EMAIL PROTECTED]>wrote:
>
> > Hi Nitin,
> > I personally do not see that as an issue. Tag <command> is meant for
> > simple sqoop command lines and there are tags <arg> for more advance use
> > cases. You happen to be in advance use case and therefore you need <arg>
> > tags. Seems a perfectly valid solution to me.
> >
> > Would you mind describe why do you necessarily need <command> tag and why
> > you can't use <arg> tags?
> >
> > Jarcec
> >
> > On Mon, Oct 22, 2012 at 10:40:18AM -0400, Nitin kak wrote:
> > > Hi Jarcec,
> > >
> > >      Thanks for ur help. Isn't that kind of a issue that we can't use
> > > "--query" in <command><command> in Oozie. If that is, can we raise this
> > > issue somewhere?
> > >
> > > Regards,
> > > Nitin Kak
> > >
> > > On Fri, Oct 19, 2012 at 5:39 PM, Jarek Jarcec Cecho <[EMAIL PROTECTED]
> > >wrote:
> > >
> > > > That is correct - you can't use <command> tag and <arg> tags together.
> > You
> > > > can use either <command> or <arg>.
> > > >
> > > > Jarcec
> > > >
> > > > On Fri, Oct 19, 2012 at 05:32:12PM -0400, Nitin kak wrote:
> > > > > Ahh...that sucks..I hope they will come up with something in future
> > > > > releases...
> > > > >
> > > > > One follow up question...We can't use <command></command> in
> > combination
> > > > > with <arg></arg> right?
> > > > >
> > > > > On Fri, Oct 19, 2012 at 5:25 PM, Jarek Jarcec Cecho <
> > [EMAIL PROTECTED]
> > > > >wrote:
> > > > >
> > > > > > Hi Nikin,
> > > > > > workaround to use --query parameter inside <command> tag in Oozie
> > is to
> > > > > > use multiple <arg> tags instead :-)
> > > > > >
> > > > > > Basically Oozie will blindly divide value supplied in <command>
> > tag by
> > > > > > spaces without taking into account any escaping. You are required
> > to
> > > > use
> > > > > > <arg> tags instead of <command> in case that you need to specify
> > > > arguments
> > > > > > with spaces. For more information please consider Oozie Sqoop
> > action
> > > > > > documentation [1].
> > > > > >
> > > > > > Jarcec
> > > > > >
> > > > > > Links:
> > > > > > 1:
> > > > > >
> > > >
> > http://oozie.apache.org/docs/3.2.0-incubating/DG_SqoopActionExtension.html
> > > > > >
> > > > > > On Fri, Oct 19, 2012 at 05:10:07PM -0400, Nitin kak wrote:
> > > > > > > Hi guys,
> > > > > > >
> > > > > > >       Just wanted to know how we can use "--query" attribute in
> > > > > > > <command></command> of sqoop action in Oozie. The issue is that
> > if we
> > > > > > have
> > > > > > > something like
> > > > > > >
> > > > > > > * <command>blah blah ....  --query "select * from
> > table"</command>
> > > > *is
> > > > > > > converted to
> > > > > > >
> > > > > > > <arg>blah</arg>
> > > > > > > <arg>blah</arg>
> > > > > > > <arg>--query</arg>
> > > > > > > <arg>"select</arg>
> > > > > > > <arg>*</arg>
> > >

Yup. Its possible.It might sound naive but can we use double quotes(") e.g
to specify that the string that follows "--query" and is in double quotes
is just one parameter?

On Mon, Oct 22, 2012 at 11:40 AM, Jarek Jarcec Cecho <[EMAIL PROTECTED]>wrote:

> Hi Nitin,
> it can actually get quite tricky to correctly split the command line when
> Oozie would allow all soft of different escaping inside <command> argument.
> That's why it fits for simple use cases and that's why there is tag <arg>
> for more advance use cases.
>
> I can see why you need to generate your whole command line in your use
> case. I've actually done something similar (conditional sqoop command line
> generation) in the past, but I've used <arg> tags and just filled argument
> values (password, username, query). Is something like that possible in your
> use case?
>
> Jarcec
>
> On Mon, Oct 22, 2012 at 11:11:21AM -0400, Nitin kak wrote:
> > Okay. So the thing is that my pre-compiler could have generated the whole
> > command(which requires the --query "select a, b as c from .... " ) as a
> > parameter. While what I now need is many more parameter values for
> > --connect, --query, --username, --password and all.
> >
> > What I feel is that if sqoop command supports "--query" construct, then
> we
> > should be able to use that construct in <command></command> in Oozie.
> >
> > Please let me know what you think.
> >
> > Regards,
> > Nitin
> >
> > On Mon, Oct 22, 2012 at 11:01 AM, Jarek Jarcec Cecho <[EMAIL PROTECTED]
> >wrote:
> >
> > > Hi Nitin,
> > > I personally do not see that as an issue. Tag <command> is meant for
> > > simple sqoop command lines and there are tags <arg> for more advance
> use
> > > cases. You happen to be in advance use case and therefore you need
> <arg>
> > > tags. Seems a perfectly valid solution to me.
> > >
> > > Would you mind describe why do you necessarily need <command> tag and
> why
> > > you can't use <arg> tags?
> > >
> > > Jarcec
> > >
> > > On Mon, Oct 22, 2012 at 10:40:18AM -0400, Nitin kak wrote:
> > > > Hi Jarcec,
> > > >
> > > >      Thanks for ur help. Isn't that kind of a issue that we can't use
> > > > "--query" in <command><command> in Oozie. If that is, can we raise
> this
> > > > issue somewhere?
> > > >
> > > > Regards,
> > > > Nitin Kak
> > > >
> > > > On Fri, Oct 19, 2012 at 5:39 PM, Jarek Jarcec Cecho <
> [EMAIL PROTECTED]
> > > >wrote:
> > > >
> > > > > That is correct - you can't use <command> tag and <arg> tags
> together.
> > > You
> > > > > can use either <command> or <arg>.
> > > > >
> > > > > Jarcec
> > > > >
> > > > > On Fri, Oct 19, 2012 at 05:32:12PM -0400, Nitin kak wrote:
> > > > > > Ahh...that sucks..I hope they will come up with something in
> future
> > > > > > releases...
> > > > > >
> > > > > > One follow up question...We can't use <command></command> in
> > > combination
> > > > > > with <arg></arg> right?
> > > > > >
> > > > > > On Fri, Oct 19, 2012 at 5:25 PM, Jarek Jarcec Cecho <
> > > [EMAIL PROTECTED]
> > > > > >wrote:
> > > > > >
> > > > > > > Hi Nikin,
> > > > > > > workaround to use --query parameter inside <command> tag in
> Oozie
> > > is to
> > > > > > > use multiple <arg> tags instead :-)
> > > > > > >
> > > > > > > Basically Oozie will blindly divide value supplied in <command>
> > > tag by
> > > > > > > spaces without taking into account any escaping. You are
> required
> > > to
> > > > > use
> > > > > > > <arg> tags instead of <command> in case that you need to
> specify
> > > > > arguments
> > > > > > > with spaces. For more information please consider Oozie Sqoop
> > > action
> > > > > > > documentation [1].
> > > > > > >
> > > > > > > Jarcec
> > > > > > >
> > > > > > > Links:
> > > > > > > 1:
> > > > > > >
> > > > >
> > >
> http://oozie.apache.org/docs/3.2.0-incubating/DG_SqoopActionExtension.html
> > > > > > >
> > > > > > > On Fri, Oct 19, 2012 at 05:10:07PM -0400, Nitin kak wrote:
> > > > > > > > Hi guys,
> > > > > > > >
> > > > > > > >       Just wanted to know how we can use "--query" attribute

Unfortunately escaping with double quotes in <command> tag is not supported. You really need to use <arg> tag where the escaping do not make sense as each argument is in it's own tag. For example:

...
  <arg>--query</arg>
  <arg>select * from my_cool_table where $CONDITIONS</arg>
...

Jarcec

On Mon, Oct 22, 2012 at 01:03:25PM -0400, Nitin kak wrote:
> Yup. Its possible.It might sound naive but can we use double quotes(") e.g
> to specify that the string that follows "--query" and is in double quotes
> is just one parameter?
>
> On Mon, Oct 22, 2012 at 11:40 AM, Jarek Jarcec Cecho <[EMAIL PROTECTED]>wrote:
>
> > Hi Nitin,
> > it can actually get quite tricky to correctly split the command line when
> > Oozie would allow all soft of different escaping inside <command> argument.
> > That's why it fits for simple use cases and that's why there is tag <arg>
> > for more advance use cases.
> >
> > I can see why you need to generate your whole command line in your use
> > case. I've actually done something similar (conditional sqoop command line
> > generation) in the past, but I've used <arg> tags and just filled argument
> > values (password, username, query). Is something like that possible in your
> > use case?
> >
> > Jarcec
> >
> > On Mon, Oct 22, 2012 at 11:11:21AM -0400, Nitin kak wrote:
> > > Okay. So the thing is that my pre-compiler could have generated the whole
> > > command(which requires the --query "select a, b as c from .... " ) as a
> > > parameter. While what I now need is many more parameter values for
> > > --connect, --query, --username, --password and all.
> > >
> > > What I feel is that if sqoop command supports "--query" construct, then
> > we
> > > should be able to use that construct in <command></command> in Oozie.
> > >
> > > Please let me know what you think.
> > >
> > > Regards,
> > > Nitin
> > >
> > > On Mon, Oct 22, 2012 at 11:01 AM, Jarek Jarcec Cecho <[EMAIL PROTECTED]
> > >wrote:
> > >
> > > > Hi Nitin,
> > > > I personally do not see that as an issue. Tag <command> is meant for
> > > > simple sqoop command lines and there are tags <arg> for more advance
> > use
> > > > cases. You happen to be in advance use case and therefore you need
> > <arg>
> > > > tags. Seems a perfectly valid solution to me.
> > > >
> > > > Would you mind describe why do you necessarily need <command> tag and
> > why
> > > > you can't use <arg> tags?
> > > >
> > > > Jarcec
> > > >
> > > > On Mon, Oct 22, 2012 at 10:40:18AM -0400, Nitin kak wrote:
> > > > > Hi Jarcec,
> > > > >
> > > > >      Thanks for ur help. Isn't that kind of a issue that we can't use
> > > > > "--query" in <command><command> in Oozie. If that is, can we raise
> > this
> > > > > issue somewhere?
> > > > >
> > > > > Regards,
> > > > > Nitin Kak
> > > > >
> > > > > On Fri, Oct 19, 2012 at 5:39 PM, Jarek Jarcec Cecho <
> > [EMAIL PROTECTED]
> > > > >wrote:
> > > > >
> > > > > > That is correct - you can't use <command> tag and <arg> tags
> > together.
> > > > You
> > > > > > can use either <command> or <arg>.
> > > > > >
> > > > > > Jarcec
> > > > > >
> > > > > > On Fri, Oct 19, 2012 at 05:32:12PM -0400, Nitin kak wrote:
> > > > > > > Ahh...that sucks..I hope they will come up with something in
> > future
> > > > > > > releases...
> > > > > > >
> > > > > > > One follow up question...We can't use <command></command> in
> > > > combination
> > > > > > > with <arg></arg> right?
> > > > > > >
> > > > > > > On Fri, Oct 19, 2012 at 5:25 PM, Jarek Jarcec Cecho <
> > > > [EMAIL PROTECTED]
> > > > > > >wrote:
> > > > > > >
> > > > > > > > Hi Nikin,
> > > > > > > > workaround to use --query parameter inside <command> tag in
> > Oozie
> > > > is to
> > > > > > > > use multiple <arg> tags instead :-)
> > > > > > > >
> > > > > > > > Basically Oozie will blindly divide value supplied in <command>
> > > > tag by
> > > > > > > > spaces without taking into account any escaping. You are
> > required
> > > > to
> > > > > > use
> > > > > > > > <arg> tags instead of <command> in case that you need to