Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Sqoop, mail # user - Sqoop incremental import ( can any just help me out)


Copy link to this message
-
Re: Sqoop incremental import ( can any just help me out)
yogesh kumar 2013-12-30, 19:27
Thanks Chalcy, I got your point, let me try a simple test for it..   but
the situation here is for incremental import i have to change the column
for split by

Its a kind of risk..   can not take a chance.  just want to be sure that.

it will not affect the hive table and data into it after
being incremental import. my incremental  import will directly pull data
and put it at where my old sqooped data resides

Want suggestion from champions of sqoop
Pls hep me out

On Tue, Dec 31, 2013 at 12:30 AM, Chalcy <[EMAIL PROTECTED]> wrote:

> I have not tried this but I believe you can change the split by as you
> wish.  The split by is used to split the jobs while --check-column and
> --last-value are used for incremental import.
>
> I do not know exact scenario but if empno gives a better split, you still
> can use that for incremental import instead of changing the split-by field.
>
> I would suggest you do a very simple test to find out.
>
> Hope this helps,
> Chalcy
>
>
> On Mon, Dec 30, 2013 at 1:18 PM, yogesh kumar <[EMAIL PROTECTED]>wrote:
>
>> Hello all,
>>
>> I have done sqoop import for a particluar table first time say table
>> Employee..
>>
>> sqoop import -libjars .....
>> --query "select empno, name, date, loc from table Employee where
>> \$CONDITIONS ..  "
>> *--split-by empno*
>> --fields-terminated-by ','
>> .
>> .
>> .
>> .
>>
>> I have created an external table on hive,
>>
>> *Now I want to pull data on daily basis by using incremental pull.  can
>> I specify the different column for --split-by*
>>
>> like
>>
>> sqoop import -libjars .....
>> --query "select empno, name, date, loc from table Employee where
>> \$CONDITIONS ..  "
>> --check-column date
>> --incremental append
>> --last-value 2013-05-01
>> *--split-by date*
>> --split-by empno
>>
>>
>> Can I change the column for *split by in incremental sqoop*, if not then
>> how to do it.
>>
>> Pls suggest
>>
>
>