Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Sqoop >> mail # user >> Sqoop import when table schema changes


Copy link to this message
-
Re: Sqoop import when table schema changes
Yes, that is the easy way if table is not too big.
On Tue, Jan 28, 2014 at 9:29 AM, Krishna Rao <[EMAIL PROTECTED]> wrote:

> Thank you for your reply.
>
> I think we'll probably just go for a simple solution: drop the table in
> hive and then re-sqoop it in each time.
>
>
> On 28 January 2014 14:17, Chalcy <[EMAIL PROTECTED]> wrote:
>
>> Yes, sqoop will not reflect schema changes.
>>
>> There are two ways you can handle,
>> 1. Simple way is to add columns to the hive table and sqoop data into it.
>>  In that case always the db columns need to be added to the end of the
>> table.  (there is a way to handle if datatype of a column changed or column
>> added in the middle)
>> 2. If you have to handle schema changes on many tables, then you have to
>> write a process to automatically find the schema changes and reflect that
>> on the hive table.  This is a multi step process.
>>
>>
>> On Tue, Jan 28, 2014 at 8:04 AM, Krishna Rao <[EMAIL PROTECTED]>wrote:
>>
>>> We have a task to sqoop-import tables from a postgreSQL db. With
>>> "--hive-overwrite" the data correctly gets refreshed. However, if a schema
>>> change occurs on the original table, it appears that the schema does not
>>> get refreshed.
>>>
>>> Is there a way to get sqoop to re-create the table if the schema has
>>> changed? Or will I need to delete the table beforehand?
>>>
>>> Krishna
>>>
>>
>>
>