Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Sqoop >> mail # user >> RE: Incremental Import for Sqoop2


Copy link to this message
-
RE: Incremental Import for Sqoop2
I faced the same issue, I ended up by running a shell-action (through oozie) after each execution of sqoop.

Inside my option parameter file I had something like this:

--whereID >= A and ID <= B

After each execution of sqoop, I run a shell script which modifies the "where" clause parameter by the chunk size based on my need.
So for each sqoop run, my option parameter becomes:
run:1--whereID >= 1 and ID <= 20
run:2--whereID >= 21 and ID <= 40
run:3--whereID >= 41 and ID <= 60
where chunk size = 20.
It worked for me!

- Tanzir

Date: Wed, 15 May 2013 00:02:51 +0530
Subject: Re: Fwd: Incremental Import for Sqoop2
From: [EMAIL PROTECTED]
To: [EMAIL PROTECTED]

Hi,
Try using Boundary query or sql statement while creating import job for filtering records.

For example: Importing id greater than 100
Boundary query: select min(id),max(id) from table where id>100
Sorry if I am wrong.
On Tue, May 14, 2013 at 11:06 PM, Joanne Chan <[EMAIL PROTECTED]> wrote:

Hi Jarek,

Thanks for the speedy response!

Any idea what the timeline is for this feature?
I am trying to utilize sqoop2 to pull new/updated data from mysql to hdfs.
Do you know of any workarounds for sqoop2 to get this job done without having to use incremental import?

Appreciate your input!

On Tue, May 14, 2013 at 1:23 PM, Jarek Jarcec Cecho <[EMAIL PROTECTED]> wrote:

Hi sir,

Sqoop 2 currently can't do incremental imports, however implementing this feature is definitely in our plan!

Jarcec

On Tue, May 14, 2013 at 01:14:32PM -0400, Joanne Chan wrote:

> Is incremental import available for sqoop2?  If not, is that part of the

> plan?

>

> I can't seem to find any good example of how to create import job with

> incremental parameter.

>

> I am on:

> client version:

>   Sqoop 2.0.0-SNAPSHOT revision c0a43d436fdd9293bf10e4b874d1002441c056b1

> server version:

>   Sqoop 2.0.0-SNAPSHOT revision c0a43d436fdd9293bf10e4b874d1002441c056b1

--
-- JChan

--
Regards
Vasanth kumar RJ