Hi guys,
     Is there a way to replace (or drop) a character in one of the fields
on the fly while importing data from an RDBMS system? I basically want to
replace Ctrl-A characters in the fields. I know its possible to do that
with Sqoop Hive Import(--hive-delims-replacement) but Hive Sqoop import is
know not to work correctly with Oozie(hope this one gets remedied soon).

Regards,
Nitin Kak

Do it first with Python??
From: Nitin kak [mailto:[EMAIL PROTECTED]]
Sent: Monday, February 04, 2013 2:42 PM
To: [EMAIL PROTECTED]
Subject: String replace functionality in Sqoop import using Oozie

Hi guys,
     Is there a way to replace (or drop) a character in one of the fields on the fly while importing data from an RDBMS system? I basically want to replace Ctrl-A characters in the fields. I know its possible to do that with Sqoop Hive Import(--hive-delims-replacement) but Hive Sqoop import is know not to work correctly with Oozie(hope this one gets remedied soon).

Regards,
Nitin Kak

Didn't get you. Could you please elaborate just a bit?

On Mon, Feb 4, 2013 at 2:43 PM, Connell, Chuck <[EMAIL PROTECTED]>wrote:

>  Do it first with Python??****
>
> ** **
>
> ** **
>
> *From:* Nitin kak [mailto:[EMAIL PROTECTED]]
> *Sent:* Monday, February 04, 2013 2:42 PM
> *To:* [EMAIL PROTECTED]
> *Subject:* String replace functionality in Sqoop import using Oozie****
>
> ** **
>
> Hi guys,****
>
> ** **
>
> ** **
>
>      Is there a way to replace (or drop) a character in one of the fields
> on the fly while importing data from an RDBMS system? I basically want to
> replace Ctrl-A characters in the fields. I know its possible to do that
> with Sqoop Hive Import(--hive-delims-replacement) but Hive Sqoop import
> is know not to work correctly with Oozie(hope this one gets remedied soon).
> ****
>
> ** **
>
> Regards,****
>
> Nitin Kak****
>

I mean to add a step, putting a Python piped script in the middle. So instead of

RDBMS --> Sqoop --> Hive

You would do

RDBMS --> some file format --> Python (replacing ctrl-a) --> modified files --> Sqoop --> Hive

Does that help?

Chuck
From: Nitin kak [mailto:[EMAIL PROTECTED]]
Sent: Monday, February 04, 2013 2:53 PM
To: [EMAIL PROTECTED]
Subject: Re: String replace functionality in Sqoop import using Oozie

Didn't get you. Could you please elaborate just a bit?
On Mon, Feb 4, 2013 at 2:43 PM, Connell, Chuck <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:
Do it first with Python??
From: Nitin kak [mailto:[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>]
Sent: Monday, February 04, 2013 2:42 PM
To: [EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>
Subject: String replace functionality in Sqoop import using Oozie

Hi guys,
     Is there a way to replace (or drop) a character in one of the fields on the fly while importing data from an RDBMS system? I basically want to replace Ctrl-A characters in the fields. I know its possible to do that with Sqoop Hive Import(--hive-delims-replacement) but Hive Sqoop import is know not to work correctly with Oozie(hope this one gets remedied soon).

Regards,
Nitin Kak

Hi Nitin,
even though that the name might suggest that command line argument --hive-delims-replacement is related to hive, it's not true and this argument is independent. Thus you can use it without --hive-import and it should work correctly in oozie as well.

Jarcec

On Mon, Feb 04, 2013 at 02:41:36PM -0500, Nitin kak wrote:
> Hi guys,
>
>
>      Is there a way to replace (or drop) a character in one of the fields
> on the fly while importing data from an RDBMS system? I basically want to
> replace Ctrl-A characters in the fields. I know its possible to do that
> with Sqoop Hive Import(--hive-delims-replacement) but Hive Sqoop import is
> know not to work correctly with Oozie(hope this one gets remedied soon).
>
> Regards,
> Nitin Kak

Ahh..Thats just perfect..thanks Jarcec

On Mon, Feb 4, 2013 at 3:28 PM, Jarek Jarcec Cecho <[EMAIL PROTECTED]>wrote:

> Hi Nitin,
> even though that the name might suggest that command line argument
> --hive-delims-replacement is related to hive, it's not true and this
> argument is independent. Thus you can use it without --hive-import and it
> should work correctly in oozie as well.
>
> Jarcec
>
> On Mon, Feb 04, 2013 at 02:41:36PM -0500, Nitin kak wrote:
> > Hi guys,
> >
> >
> >      Is there a way to replace (or drop) a character in one of the fields
> > on the fly while importing data from an RDBMS system? I basically want to
> > replace Ctrl-A characters in the fields. I know its possible to do that
> > with Sqoop Hive Import(--hive-delims-replacement) but Hive Sqoop import
> is
> > know not to work correctly with Oozie(hope this one gets remedied soon).
> >
> > Regards,
> > Nitin Kak
>

Hi Jarcec,

      Its works perfect. Thanks. Do you think the
name--hive-delims-replacement is a little misguiding, in terms of the scope
it can be used with. I feel that for both parameters

--hive-drop-import-delims and
--hive-delims-replacement

Should we raise a ticket to change the names to reflect the scope of these
clauses in the future release(like probably remove "hive" from them).

Thanks,
Nitin
On Mon, Feb 4, 2013 at 3:31 PM, Nitin kak <[EMAIL PROTECTED]> wrote:

> Ahh..Thats just perfect..thanks Jarcec
>
>
> On Mon, Feb 4, 2013 at 3:28 PM, Jarek Jarcec Cecho <[EMAIL PROTECTED]>wrote:
>
>> Hi Nitin,
>> even though that the name might suggest that command line argument
>> --hive-delims-replacement is related to hive, it's not true and this
>> argument is independent. Thus you can use it without --hive-import and it
>> should work correctly in oozie as well.
>>
>> Jarcec
>>
>> On Mon, Feb 04, 2013 at 02:41:36PM -0500, Nitin kak wrote:
>> > Hi guys,
>> >
>> >
>> >      Is there a way to replace (or drop) a character in one of the
>> fields
>> > on the fly while importing data from an RDBMS system? I basically want
>> to
>> > replace Ctrl-A characters in the fields. I know its possible to do that
>> > with Sqoop Hive Import(--hive-delims-replacement) but Hive Sqoop import
>> is
>> > know not to work correctly with Oozie(hope this one gets remedied soon).
>> >
>> > Regards,
>> > Nitin Kak
>>
>
>

Hi Nitin,
I'm glad to hear that it works for you! Thanks for suggesting the rename, unfortunately I'm afraid that we can't simply rename those arguments as we would break backward compatibility.

Jarcec

On Mon, Feb 04, 2013 at 05:39:13PM -0500, Nitin kak wrote:
> Hi Jarcec,
>
>       Its works perfect. Thanks. Do you think the
> name--hive-delims-replacement is a little misguiding, in terms of the scope
> it can be used with. I feel that for both parameters
>
> --hive-drop-import-delims and
> --hive-delims-replacement
>
> Should we raise a ticket to change the names to reflect the scope of these
> clauses in the future release(like probably remove "hive" from them).
>
> Thanks,
> Nitin
>
>
> On Mon, Feb 4, 2013 at 3:31 PM, Nitin kak <[EMAIL PROTECTED]> wrote:
>
> > Ahh..Thats just perfect..thanks Jarcec
> >
> >
> > On Mon, Feb 4, 2013 at 3:28 PM, Jarek Jarcec Cecho <[EMAIL PROTECTED]>wrote:
> >
> >> Hi Nitin,
> >> even though that the name might suggest that command line argument
> >> --hive-delims-replacement is related to hive, it's not true and this
> >> argument is independent. Thus you can use it without --hive-import and it
> >> should work correctly in oozie as well.
> >>
> >> Jarcec
> >>
> >> On Mon, Feb 04, 2013 at 02:41:36PM -0500, Nitin kak wrote:
> >> > Hi guys,
> >> >
> >> >
> >> >      Is there a way to replace (or drop) a character in one of the
> >> fields
> >> > on the fly while importing data from an RDBMS system? I basically want
> >> to
> >> > replace Ctrl-A characters in the fields. I know its possible to do that
> >> > with Sqoop Hive Import(--hive-delims-replacement) but Hive Sqoop import
> >> is
> >> > know not to work correctly with Oozie(hope this one gets remedied soon).
> >> >
> >> > Regards,
> >> > Nitin Kak
> >>
> >
> >

Isn't this exactly the case for deprecation with a warning and eventual
phase-out?

On Mon, Feb 4, 2013 at 5:10 PM, Jarek Jarcec Cecho <[EMAIL PROTECTED]>wrote:

> Hi Nitin,
> I'm glad to hear that it works for you! Thanks for suggesting the rename,
> unfortunately I'm afraid that we can't simply rename those arguments as we
> would break backward compatibility.
>
> Jarcec
>
> On Mon, Feb 04, 2013 at 05:39:13PM -0500, Nitin kak wrote:
> > Hi Jarcec,
> >
> >       Its works perfect. Thanks. Do you think the
> > name--hive-delims-replacement is a little misguiding, in terms of the
> scope
> > it can be used with. I feel that for both parameters
> >
> > --hive-drop-import-delims and
> > --hive-delims-replacement
> >
> > Should we raise a ticket to change the names to reflect the scope of
> these
> > clauses in the future release(like probably remove "hive" from them).
> >
> > Thanks,
> > Nitin
> >
> >
> > On Mon, Feb 4, 2013 at 3:31 PM, Nitin kak <[EMAIL PROTECTED]> wrote:
> >
> > > Ahh..Thats just perfect..thanks Jarcec
> > >
> > >
> > > On Mon, Feb 4, 2013 at 3:28 PM, Jarek Jarcec Cecho <[EMAIL PROTECTED]
> >wrote:
> > >
> > >> Hi Nitin,
> > >> even though that the name might suggest that command line argument
> > >> --hive-delims-replacement is related to hive, it's not true and this
> > >> argument is independent. Thus you can use it without --hive-import
> and it
> > >> should work correctly in oozie as well.
> > >>
> > >> Jarcec
> > >>
> > >> On Mon, Feb 04, 2013 at 02:41:36PM -0500, Nitin kak wrote:
> > >> > Hi guys,
> > >> >
> > >> >
> > >> >      Is there a way to replace (or drop) a character in one of the
> > >> fields
> > >> > on the fly while importing data from an RDBMS system? I basically
> want
> > >> to
> > >> > replace Ctrl-A characters in the fields. I know its possible to do
> that
> > >> > with Sqoop Hive Import(--hive-delims-replacement) but Hive Sqoop
> import
> > >> is
> > >> > know not to work correctly with Oozie(hope this one gets remedied
> soon).
> > >> >
> > >> > Regards,
> > >> > Nitin Kak
> > >>
> > >
> > >
>

Hi Mason,
thank you very much for your comment. I completely agree normal approach would be to mark those parameters as deprecated in next major release (for example 1.5.0) and completely remove them on subsequent major release (like 1.6.0). However right now, we're keeping Sqoop 1 in version 1.4.x and instead of working on major release 1.5, we're focusing our effort on Sqoop 2. So I'm not sure if marking those parameters deprecated would actually make sense as I'm not expecting that we will release 1.5.0 nor 1.6.0.

Jarcec

On Wed, Feb 06, 2013 at 09:38:08AM -0800, Mason wrote:
> Isn't this exactly the case for deprecation with a warning and eventual
> phase-out?
>
> On Mon, Feb 4, 2013 at 5:10 PM, Jarek Jarcec Cecho <[EMAIL PROTECTED]>wrote:
>
> > Hi Nitin,
> > I'm glad to hear that it works for you! Thanks for suggesting the rename,
> > unfortunately I'm afraid that we can't simply rename those arguments as we
> > would break backward compatibility.
> >
> > Jarcec
> >
> > On Mon, Feb 04, 2013 at 05:39:13PM -0500, Nitin kak wrote:
> > > Hi Jarcec,
> > >
> > >       Its works perfect. Thanks. Do you think the
> > > name--hive-delims-replacement is a little misguiding, in terms of the
> > scope
> > > it can be used with. I feel that for both parameters
> > >
> > > --hive-drop-import-delims and
> > > --hive-delims-replacement
> > >
> > > Should we raise a ticket to change the names to reflect the scope of
> > these
> > > clauses in the future release(like probably remove "hive" from them).
> > >
> > > Thanks,
> > > Nitin
> > >
> > >
> > > On Mon, Feb 4, 2013 at 3:31 PM, Nitin kak <[EMAIL PROTECTED]> wrote:
> > >
> > > > Ahh..Thats just perfect..thanks Jarcec
> > > >
> > > >
> > > > On Mon, Feb 4, 2013 at 3:28 PM, Jarek Jarcec Cecho <[EMAIL PROTECTED]
> > >wrote:
> > > >
> > > >> Hi Nitin,
> > > >> even though that the name might suggest that command line argument
> > > >> --hive-delims-replacement is related to hive, it's not true and this
> > > >> argument is independent. Thus you can use it without --hive-import
> > and it
> > > >> should work correctly in oozie as well.
> > > >>
> > > >> Jarcec
> > > >>
> > > >> On Mon, Feb 04, 2013 at 02:41:36PM -0500, Nitin kak wrote:
> > > >> > Hi guys,
> > > >> >
> > > >> >
> > > >> >      Is there a way to replace (or drop) a character in one of the
> > > >> fields
> > > >> > on the fly while importing data from an RDBMS system? I basically
> > want
> > > >> to
> > > >> > replace Ctrl-A characters in the fields. I know its possible to do
> > that
> > > >> > with Sqoop Hive Import(--hive-delims-replacement) but Hive Sqoop
> > import
> > > >> is
> > > >> > know not to work correctly with Oozie(hope this one gets remedied
> > soon).
> > > >> >
> > > >> > Regards,
> > > >> > Nitin Kak
> > > >>
> > > >
> > > >
> >

I see. Thanks for the explanation Jarcec.

On Wed, Feb 6, 2013 at 9:47 AM, Jarek Jarcec Cecho <[EMAIL PROTECTED]>wrote:

> Hi Mason,
> thank you very much for your comment. I completely agree normal approach
> would be to mark those parameters as deprecated in next major release (for
> example 1.5.0) and completely remove them on subsequent major release (like
> 1.6.0). However right now, we're keeping Sqoop 1 in version 1.4.x and
> instead of working on major release 1.5, we're focusing our effort on Sqoop
> 2. So I'm not sure if marking those parameters deprecated would actually
> make sense as I'm not expecting that we will release 1.5.0 nor 1.6.0.
>
> Jarcec
>
> On Wed, Feb 06, 2013 at 09:38:08AM -0800, Mason wrote:
> > Isn't this exactly the case for deprecation with a warning and eventual
> > phase-out?
> >
> > On Mon, Feb 4, 2013 at 5:10 PM, Jarek Jarcec Cecho <[EMAIL PROTECTED]
> >wrote:
> >
> > > Hi Nitin,
> > > I'm glad to hear that it works for you! Thanks for suggesting the
> rename,
> > > unfortunately I'm afraid that we can't simply rename those arguments
> as we
> > > would break backward compatibility.
> > >
> > > Jarcec
> > >
> > > On Mon, Feb 04, 2013 at 05:39:13PM -0500, Nitin kak wrote:
> > > > Hi Jarcec,
> > > >
> > > >       Its works perfect. Thanks. Do you think the
> > > > name--hive-delims-replacement is a little misguiding, in terms of the
> > > scope
> > > > it can be used with. I feel that for both parameters
> > > >
> > > > --hive-drop-import-delims and
> > > > --hive-delims-replacement
> > > >
> > > > Should we raise a ticket to change the names to reflect the scope of
> > > these
> > > > clauses in the future release(like probably remove "hive" from them).
> > > >
> > > > Thanks,
> > > > Nitin
> > > >
> > > >
> > > > On Mon, Feb 4, 2013 at 3:31 PM, Nitin kak <[EMAIL PROTECTED]>
> wrote:
> > > >
> > > > > Ahh..Thats just perfect..thanks Jarcec
> > > > >
> > > > >
> > > > > On Mon, Feb 4, 2013 at 3:28 PM, Jarek Jarcec Cecho <
> [EMAIL PROTECTED]
> > > >wrote:
> > > > >
> > > > >> Hi Nitin,
> > > > >> even though that the name might suggest that command line argument
> > > > >> --hive-delims-replacement is related to hive, it's not true and
> this
> > > > >> argument is independent. Thus you can use it without --hive-import
> > > and it
> > > > >> should work correctly in oozie as well.
> > > > >>
> > > > >> Jarcec
> > > > >>
> > > > >> On Mon, Feb 04, 2013 at 02:41:36PM -0500, Nitin kak wrote:
> > > > >> > Hi guys,
> > > > >> >
> > > > >> >
> > > > >> >      Is there a way to replace (or drop) a character in one of
> the
> > > > >> fields
> > > > >> > on the fly while importing data from an RDBMS system? I
> basically
> > > want
> > > > >> to
> > > > >> > replace Ctrl-A characters in the fields. I know its possible to
> do
> > > that
> > > > >> > with Sqoop Hive Import(--hive-delims-replacement) but Hive Sqoop
> > > import
> > > > >> is
> > > > >> > know not to work correctly with Oozie(hope this one gets
> remedied
> > > soon).
> > > > >> >
> > > > >> > Regards,
> > > > >> > Nitin Kak
> > > > >>
> > > > >
> > > > >
> > >
>