Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Sqoop >> mail # user >> Sqoop import when table schema changes


Copy link to this message
-
Re: Sqoop import when table schema changes
Yes, that is the easy way if table is not too big.
On Tue, Jan 28, 2014 at 9:29 AM, Krishna Rao <[EMAIL PROTECTED]> wrote:

> Thank you for your reply.
>
> I think we'll probably just go for a simple solution: drop the table in
> hive and then re-sqoop it in each time.
>
>
> On 28 January 2014 14:17, Chalcy <[EMAIL PROTECTED]> wrote:
>
>> Yes, sqoop will not reflect schema changes.
>>
>> There are two ways you can handle,
>> 1. Simple way is to add columns to the hive table and sqoop data into it.
>>  In that case always the db columns need to be added to the end of the
>> table.  (there is a way to handle if datatype of a column changed or column
>> added in the middle)
>> 2. If you have to handle schema changes on many tables, then you have to
>> write a process to automatically find the schema changes and reflect that
>> on the hive table.  This is a multi step process.
>>
>>
>> On Tue, Jan 28, 2014 at 8:04 AM, Krishna Rao <[EMAIL PROTECTED]>wrote:
>>
>>> We have a task to sqoop-import tables from a postgreSQL db. With
>>> "--hive-overwrite" the data correctly gets refreshed. However, if a schema
>>> change occurs on the original table, it appears that the schema does not
>>> get refreshed.
>>>
>>> Is there a way to get sqoop to re-create the table if the schema has
>>> changed? Or will I need to delete the table beforehand?
>>>
>>> Krishna
>>>
>>
>>
>

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB