Hi All,

     Is there a way to specify the Hive database in the "sqoop-import"
command into hive(using --hive-import). Tried --hive-table
"<database_name>.<table_name>" does not work. I ran the command using Oozie
but think wont work otherwise too.

Thanks,
Nitin

Hi Nitin,
would you mind sharing what Sqoop and Hive version are you using? The --hive-table command in form <database>.<table> is the correct and supported way. However I barely remember that I faced similar issue in the past, updating Hive has solved that for me.

Also please note that hive import is not supported when is Sqoop used from Oozie. Recommended workaround is to split the task into two actions - in first sqoop action import data into HDFS, in second hive action load them into Hive.

Jarcec

On Wed, Jan 30, 2013 at 06:00:01PM -0500, Nitin kak wrote:
> Hi All,
>
>      Is there a way to specify the Hive database in the "sqoop-import"
> command into hive(using --hive-import). Tried --hive-table
> "<database_name>.<table_name>" does not work. I ran the command using Oozie
> but think wont work otherwise too.
>
> Thanks,
> Nitin

I am using sqoop 1.4 and Hive 0.9.

"Also please note that hive import is not supported when is Sqoop used from
Oozie.". Thats strange...because I am using it.

I was doing it the way you said before but there is a "null" value issue
which comes in between. The thing is while importing to a hive table from a
HDFS directory, the NULL values are taken as null strings(well at least for
the non-numeric columns). This creates an inconsistency as for numeric
columns Hive takes \N as null while for string columns, there is "null"
string. This creates issue while exporting data back from HDFS to RDBMS.

On Wed, Jan 30, 2013 at 6:22 PM, Jarek Jarcec Cecho <[EMAIL PROTECTED]>wrote:

> Hi Nitin,
> would you mind sharing what Sqoop and Hive version are you using? The
> --hive-table command in form <database>.<table> is the correct and
> supported way. However I barely remember that I faced similar issue in the
> past, updating Hive has solved that for me.
>
> Also please note that hive import is not supported when is Sqoop used from
> Oozie. Recommended workaround is to split the task into two actions - in
> first sqoop action import data into HDFS, in second hive action load them
> into Hive.
>
> Jarcec
>
> On Wed, Jan 30, 2013 at 06:00:01PM -0500, Nitin kak wrote:
> > Hi All,
> >
> >      Is there a way to specify the Hive database in the "sqoop-import"
> > command into hive(using --hive-import). Tried --hive-table
> > "<database_name>.<table_name>" does not work. I ran the command using
> Oozie
> > but think wont work otherwise too.
> >
> > Thanks,
> > Nitin
>

I can see the --hive-import in the CDH4 documentation as well.

http://archive.cloudera.com/cdh4/cdh/4/sqoop/SqoopUserGuide.html#_literal_sqoop_import_literal

On Wed, Jan 30, 2013 at 6:33 PM, Nitin kak <[EMAIL PROTECTED]> wrote:

> I am using sqoop 1.4 and Hive 0.9.
>
> "Also please note that hive import is not supported when is Sqoop used
> from Oozie.". Thats strange...because I am using it.
>
> I was doing it the way you said before but there is a "null" value issue
> which comes in between. The thing is while importing to a hive table from a
> HDFS directory, the NULL values are taken as null strings(well at least for
> the non-numeric columns). This creates an inconsistency as for numeric
> columns Hive takes \N as null while for string columns, there is "null"
> string. This creates issue while exporting data back from HDFS to RDBMS.
>
>
> On Wed, Jan 30, 2013 at 6:22 PM, Jarek Jarcec Cecho <[EMAIL PROTECTED]>wrote:
>
>> Hi Nitin,
>> would you mind sharing what Sqoop and Hive version are you using? The
>> --hive-table command in form <database>.<table> is the correct and
>> supported way. However I barely remember that I faced similar issue in the
>> past, updating Hive has solved that for me.
>>
>> Also please note that hive import is not supported when is Sqoop used
>> from Oozie. Recommended workaround is to split the task into two actions -
>> in first sqoop action import data into HDFS, in second hive action load
>> them into Hive.
>>
>> Jarcec
>>
>> On Wed, Jan 30, 2013 at 06:00:01PM -0500, Nitin kak wrote:
>> > Hi All,
>> >
>> >      Is there a way to specify the Hive database in the "sqoop-import"
>> > command into hive(using --hive-import). Tried --hive-table
>> > "<database_name>.<table_name>" does not work. I ran the command using
>> Oozie
>> > but think wont work otherwise too.
>> >
>> > Thanks,
>> > Nitin
>>
>
>

Hi Nitin,
Sqoop will by default use string "null" as substitution for NULL values. You can override this default behaviour by using following two arguments:

  --null-string '\\N' --null-non-string '\\N'

Please note that Sqoop is using those parameters in generated source code and thus they need to be properly escaped. The example with \\N will actually results in \N being used (that is compatible with Hive). You can find more information about null substitution handling in Sqoop user guide [1].

Jarcec

Links:
1: http://s.apache.org/nq

On Wed, Jan 30, 2013 at 06:33:31PM -0500, Nitin kak wrote:
> I am using sqoop 1.4 and Hive 0.9.
>
> "Also please note that hive import is not supported when is Sqoop used from
> Oozie.". Thats strange...because I am using it.
>
> I was doing it the way you said before but there is a "null" value issue
> which comes in between. The thing is while importing to a hive table from a
> HDFS directory, the NULL values are taken as null strings(well at least for
> the non-numeric columns). This creates an inconsistency as for numeric
> columns Hive takes \N as null while for string columns, there is "null"
> string. This creates issue while exporting data back from HDFS to RDBMS.
>
> On Wed, Jan 30, 2013 at 6:22 PM, Jarek Jarcec Cecho <[EMAIL PROTECTED]>wrote:
>
> > Hi Nitin,
> > would you mind sharing what Sqoop and Hive version are you using? The
> > --hive-table command in form <database>.<table> is the correct and
> > supported way. However I barely remember that I faced similar issue in the
> > past, updating Hive has solved that for me.
> >
> > Also please note that hive import is not supported when is Sqoop used from
> > Oozie. Recommended workaround is to split the task into two actions - in
> > first sqoop action import data into HDFS, in second hive action load them
> > into Hive.
> >
> > Jarcec
> >
> > On Wed, Jan 30, 2013 at 06:00:01PM -0500, Nitin kak wrote:
> > > Hi All,
> > >
> > >      Is there a way to specify the Hive database in the "sqoop-import"
> > > command into hive(using --hive-import). Tried --hive-table
> > > "<database_name>.<table_name>" does not work. I ran the command using
> > Oozie
> > > but think wont work otherwise too.
> > >
> > > Thanks,
> > > Nitin
> >

Hi Nitin,
parameter --hive-import is supported and working correctly when you're running Sqoop from command line. It's known to not work correctly when you run Sqoop from Oozie and thus it's not supported only when running from Oozie.

The recommended workaround is to split your Sqoop action into two actions instead:

* In first Sqoop action load data into temporary directory on HDFS
* In subsequent second Hive action load data from temporary directory into Hive

Jarcec

On Wed, Jan 30, 2013 at 06:41:31PM -0500, Nitin kak wrote:
> I can see the --hive-import in the CDH4 documentation as well.
>
> http://archive.cloudera.com/cdh4/cdh/4/sqoop/SqoopUserGuide.html#_literal_sqoop_import_literal
>
> On Wed, Jan 30, 2013 at 6:33 PM, Nitin kak <[EMAIL PROTECTED]> wrote:
>
> > I am using sqoop 1.4 and Hive 0.9.
> >
> > "Also please note that hive import is not supported when is Sqoop used
> > from Oozie.". Thats strange...because I am using it.
> >
> > I was doing it the way you said before but there is a "null" value issue
> > which comes in between. The thing is while importing to a hive table from a
> > HDFS directory, the NULL values are taken as null strings(well at least for
> > the non-numeric columns). This creates an inconsistency as for numeric
> > columns Hive takes \N as null while for string columns, there is "null"
> > string. This creates issue while exporting data back from HDFS to RDBMS.
> >
> >
> > On Wed, Jan 30, 2013 at 6:22 PM, Jarek Jarcec Cecho <[EMAIL PROTECTED]>wrote:
> >
> >> Hi Nitin,
> >> would you mind sharing what Sqoop and Hive version are you using? The
> >> --hive-table command in form <database>.<table> is the correct and
> >> supported way. However I barely remember that I faced similar issue in the
> >> past, updating Hive has solved that for me.
> >>
> >> Also please note that hive import is not supported when is Sqoop used
> >> from Oozie. Recommended workaround is to split the task into two actions -
> >> in first sqoop action import data into HDFS, in second hive action load
> >> them into Hive.
> >>
> >> Jarcec
> >>
> >> On Wed, Jan 30, 2013 at 06:00:01PM -0500, Nitin kak wrote:
> >> > Hi All,
> >> >
> >> >      Is there a way to specify the Hive database in the "sqoop-import"
> >> > command into hive(using --hive-import). Tried --hive-table
> >> > "<database_name>.<table_name>" does not work. I ran the command using
> >> Oozie
> >> > but think wont work otherwise too.
> >> >
> >> > Thanks,
> >> > Nitin
> >>
> >
> >

Thanks Jarcec.

Is there a plan to support it in the future because it seems to be a pretty
obvious thing to assume that it would work?

On Thu, Jan 31, 2013 at 11:27 AM, Jarek Jarcec Cecho <[EMAIL PROTECTED]>wrote:

> Hi Nitin,
> parameter --hive-import is supported and working correctly when you're
> running Sqoop from command line. It's known to not work correctly when you
> run Sqoop from Oozie and thus it's not supported only when running from
> Oozie.
>
> The recommended workaround is to split your Sqoop action into two actions
> instead:
>
> * In first Sqoop action load data into temporary directory on HDFS
> * In subsequent second Hive action load data from temporary directory into
> Hive
>
> Jarcec
>
> On Wed, Jan 30, 2013 at 06:41:31PM -0500, Nitin kak wrote:
> > I can see the --hive-import in the CDH4 documentation as well.
> >
> >
> http://archive.cloudera.com/cdh4/cdh/4/sqoop/SqoopUserGuide.html#_literal_sqoop_import_literal
> >
> > On Wed, Jan 30, 2013 at 6:33 PM, Nitin kak <[EMAIL PROTECTED]>
> wrote:
> >
> > > I am using sqoop 1.4 and Hive 0.9.
> > >
> > > "Also please note that hive import is not supported when is Sqoop used
> > > from Oozie.". Thats strange...because I am using it.
> > >
> > > I was doing it the way you said before but there is a "null" value
> issue
> > > which comes in between. The thing is while importing to a hive table
> from a
> > > HDFS directory, the NULL values are taken as null strings(well at
> least for
> > > the non-numeric columns). This creates an inconsistency as for numeric
> > > columns Hive takes \N as null while for string columns, there is "null"
> > > string. This creates issue while exporting data back from HDFS to
> RDBMS.
> > >
> > >
> > > On Wed, Jan 30, 2013 at 6:22 PM, Jarek Jarcec Cecho <[EMAIL PROTECTED]
> >wrote:
> > >
> > >> Hi Nitin,
> > >> would you mind sharing what Sqoop and Hive version are you using? The
> > >> --hive-table command in form <database>.<table> is the correct and
> > >> supported way. However I barely remember that I faced similar issue
> in the
> > >> past, updating Hive has solved that for me.
> > >>
> > >> Also please note that hive import is not supported when is Sqoop used
> > >> from Oozie. Recommended workaround is to split the task into two
> actions -
> > >> in first sqoop action import data into HDFS, in second hive action
> load
> > >> them into Hive.
> > >>
> > >> Jarcec
> > >>
> > >> On Wed, Jan 30, 2013 at 06:00:01PM -0500, Nitin kak wrote:
> > >> > Hi All,
> > >> >
> > >> >      Is there a way to specify the Hive database in the
> "sqoop-import"
> > >> > command into hive(using --hive-import). Tried --hive-table
> > >> > "<database_name>.<table_name>" does not work. I ran the command
> using
> > >> Oozie
> > >> > but think wont work otherwise too.
> > >> >
> > >> > Thanks,
> > >> > Nitin
> > >>
> > >
> > >
>

Hi Nitin,
as there is very simple workaround, nobody seems to be working on it at the moment. We will make sure that such limitation won't be there in Sqoop 2 though.

Jarcec

On Thu, Jan 31, 2013 at 12:46:09PM -0500, Nitin kak wrote:
> Thanks Jarcec.
>
> Is there a plan to support it in the future because it seems to be a pretty
> obvious thing to assume that it would work?
>
> On Thu, Jan 31, 2013 at 11:27 AM, Jarek Jarcec Cecho <[EMAIL PROTECTED]>wrote:
>
> > Hi Nitin,
> > parameter --hive-import is supported and working correctly when you're
> > running Sqoop from command line. It's known to not work correctly when you
> > run Sqoop from Oozie and thus it's not supported only when running from
> > Oozie.
> >
> > The recommended workaround is to split your Sqoop action into two actions
> > instead:
> >
> > * In first Sqoop action load data into temporary directory on HDFS
> > * In subsequent second Hive action load data from temporary directory into
> > Hive
> >
> > Jarcec
> >
> > On Wed, Jan 30, 2013 at 06:41:31PM -0500, Nitin kak wrote:
> > > I can see the --hive-import in the CDH4 documentation as well.
> > >
> > >
> > http://archive.cloudera.com/cdh4/cdh/4/sqoop/SqoopUserGuide.html#_literal_sqoop_import_literal
> > >
> > > On Wed, Jan 30, 2013 at 6:33 PM, Nitin kak <[EMAIL PROTECTED]>
> > wrote:
> > >
> > > > I am using sqoop 1.4 and Hive 0.9.
> > > >
> > > > "Also please note that hive import is not supported when is Sqoop used
> > > > from Oozie.". Thats strange...because I am using it.
> > > >
> > > > I was doing it the way you said before but there is a "null" value
> > issue
> > > > which comes in between. The thing is while importing to a hive table
> > from a
> > > > HDFS directory, the NULL values are taken as null strings(well at
> > least for
> > > > the non-numeric columns). This creates an inconsistency as for numeric
> > > > columns Hive takes \N as null while for string columns, there is "null"
> > > > string. This creates issue while exporting data back from HDFS to
> > RDBMS.
> > > >
> > > >
> > > > On Wed, Jan 30, 2013 at 6:22 PM, Jarek Jarcec Cecho <[EMAIL PROTECTED]
> > >wrote:
> > > >
> > > >> Hi Nitin,
> > > >> would you mind sharing what Sqoop and Hive version are you using? The
> > > >> --hive-table command in form <database>.<table> is the correct and
> > > >> supported way. However I barely remember that I faced similar issue
> > in the
> > > >> past, updating Hive has solved that for me.
> > > >>
> > > >> Also please note that hive import is not supported when is Sqoop used
> > > >> from Oozie. Recommended workaround is to split the task into two
> > actions -
> > > >> in first sqoop action import data into HDFS, in second hive action
> > load
> > > >> them into Hive.
> > > >>
> > > >> Jarcec
> > > >>
> > > >> On Wed, Jan 30, 2013 at 06:00:01PM -0500, Nitin kak wrote:
> > > >> > Hi All,
> > > >> >
> > > >> >      Is there a way to specify the Hive database in the
> > "sqoop-import"
> > > >> > command into hive(using --hive-import). Tried --hive-table
> > > >> > "<database_name>.<table_name>" does not work. I ran the command
> > using
> > > >> Oozie
> > > >> > but think wont work otherwise too.
> > > >> >
> > > >> > Thanks,
> > > >> > Nitin
> > > >>
> > > >
> > > >
> >