Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hive, mail # user - add  column to hive table


Copy link to this message
-
RE: add column to hive table
Chalcy Raja 2012-05-02, 13:00
Yes, Gabi, that is the reason I went this solution as well.  Now it is hard to keep up with db changes in hive.

Question to the group: How are you all handling the ever changing schema?

Thanks,
Chalcy

From: Gabi D [mailto:[EMAIL PROTECTED]]
Sent: Wednesday, May 02, 2012 2:50 AM
To: [EMAIL PROTECTED]
Cc: [EMAIL PROTECTED]
Subject: Re: add column to hive table

you probably noticed this already, but if you add a column in the middle and it did not exist in your older files then when you select from older dates you will get wrong values in the wrong columns since you will be looking at old files with the new format. Dangerous. We also went with the sqoop trick to cause new columns to be added only to the end of existing hive tables. It causes a difference in the order of fields between the db and hive but it's not that painful.
On Wed, May 2, 2012 at 3:38 AM, Chalcy Raja <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:
In my situation, the tables I was importing into hive are daily tables.  Couple of columns were added in a month which are not added to the end of the table. Also one field got dropped in between. Also I have data for a year.

Anyway the way, I handled is, created an external table on top of the older data and sqoop imported specifying columns in the query tag of the sqoop import into the hive external table.  Had to have three different sqoop imports for the three periods.  Anyway got them all together so nobody will notice the change :)

--Chalcy

From: Chalcy Raja [mailto:[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>]
Sent: Monday, April 30, 2012 9:08 PM
To: [EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>; [EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>
Subject: RE: add column to hive table

Good idea! Ashwanth.  See if this can work for me and let you know.

Thanks,
CHalcy

From: [EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]> [mailto:[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>] On Behalf Of Ashwanth Kumar
Sent: Monday, April 30, 2012 6:55 PM
To: [EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>
Subject: Re: add column to hive table

You can only add columns at the end. If you are using the native SerDe try

ALTER TABLE tbl_name REPLACE COLUMNS (column1 int, mynewcolumn string, column2 int);

More information on Hive Wiki - https://cwiki.apache.org/Hive/languagemanual-ddl.html#LanguageManualDDL-Add%252FReplaceColumns
On Tue, May 1, 2012 at 4:12 AM, Chalcy Raja <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:
Hi all,
Adding a column by alter table mytable add column (mynewcolumn string); works well for me.  This adds a field to the end of the schema.
Is there a way to add a column in between two columns?
Thanks,
Chalcy

--

Ashwanth Kumar / ashwanthkumar.in<http://ashwanthkumar.in/>