Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hive, mail # dev - Review Request 14169: HIVE-3764: Support metastore version consistency check


Copy link to this message
-
Re: Review Request 14169: HIVE-3764: Support metastore version consistency check
Prasad Mujumdar 2013-09-18, 05:45

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14169/
-----------------------------------------------------------

(Updated Sept. 18, 2013, 5:45 a.m.)
Review request for hive, Ashutosh Chauhan and Brock Noland.
Changes
-------

Changes per review feedback.
- rename the comment field.
- fixed typos and updated variable name
- fixed the version comparison case
- build script update to set the hive dist path for testing (was missed in the original diff when the large patch was split into two.)
Bugs: HIVE-3764
    https://issues.apache.org/jira/browse/HIVE-3764
Repository: hive-git
Description
-------

This is a 0.12 specific patch. The trunk patch will include additional metastore scripts which I will attach separately to the ticket.

- Added a new table in the metastore schema to store the Hive version in the metastore.
- Metastore handler compare the version stored in the schema with its own version. If there's a mismatch, then it can either record the correct version or raise error. The behavior is configurable via a new Hive config. This config when set, also restrict dataNucleus to auto upgrade the schema.
- The new schema creation and upgrade scripts record the new version in the metastore version table.
- Added 0.12 upgrade scripts for all supported DBs to creates the new table version tables in 0.12 metastore schema

The current patch has the verification turned off by default. I would prefer to keep it enabled, though it require any add-hoc setup to explicitly disable it (or create the metastore schema by running scripts). The default can be changed or left as is as per the consensus.
Diffs (updated)
-----

  build-common.xml ad5ac23
  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1474469
  conf/hive-default.xml.template c123452
  metastore/scripts/upgrade/derby/014-HIVE-3764.derby.sql PRE-CREATION
  metastore/scripts/upgrade/derby/hive-schema-0.12.0.derby.sql cce544f
  metastore/scripts/upgrade/derby/upgrade-0.10.0-to-0.11.0.derby.sql cae7936
  metastore/scripts/upgrade/derby/upgrade-0.11.0-to-0.12.0.derby.sql 492cc93
  metastore/scripts/upgrade/derby/upgrade.order.derby PRE-CREATION
  metastore/scripts/upgrade/mysql/014-HIVE-3764.mysql.sql PRE-CREATION
  metastore/scripts/upgrade/mysql/hive-schema-0.12.0.mysql.sql 22a77fe
  metastore/scripts/upgrade/mysql/upgrade-0.11.0-to-0.12.0.mysql.sql 375a05f
  metastore/scripts/upgrade/mysql/upgrade.order.mysql PRE-CREATION
  metastore/scripts/upgrade/oracle/014-HIVE-3764.oracle.sql PRE-CREATION
  metastore/scripts/upgrade/oracle/hive-schema-0.12.0.oracle.sql 85a0178
  metastore/scripts/upgrade/oracle/upgrade-0.10.0-to-0.11.0.mysql.sql PRE-CREATION
  metastore/scripts/upgrade/oracle/upgrade-0.11.0-to-0.12.0.oracle.sql a2d0901
  metastore/scripts/upgrade/oracle/upgrade.order.oracle PRE-CREATION
  metastore/scripts/upgrade/postgres/014-HIVE-3764.postgres.sql PRE-CREATION
  metastore/scripts/upgrade/postgres/hive-schema-0.12.0.postgres.sql 7b319ba
  metastore/scripts/upgrade/postgres/upgrade-0.11.0-to-0.12.0.postgres.sql 9da0a1b
  metastore/scripts/upgrade/postgres/upgrade.order.postgres PRE-CREATION
  metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 39dda92
  metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreSchemaInfo.java PRE-CREATION
  metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java a27243d
  metastore/src/java/org/apache/hadoop/hive/metastore/RawStore.java e410c3a
  metastore/src/model/org/apache/hadoop/hive/metastore/model/MVersionTable.java PRE-CREATION
  metastore/src/model/package.jdo c42b5b0
  metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java 8066784
  metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java 0f9b16c
  metastore/src/test/org/apache/hadoop/hive/metastore/TestMetastoreVersion.java PRE-CREATION

Diff: https://reviews.apache.org/r/14169/diff/
Testing

Added new tests for schema verification. Manually tested the upgrades using derby and MySQL.
Thanks,

Prasad Mujumdar