Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Drill, mail # dev - Review Request 17626: DRILL-354: Hive storage engine - Phase 1


Copy link to this message
-
Review Request 17626: DRILL-354: Hive storage engine - Phase 1
Venki Korukanti 2014-01-31, 22:53

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/17626/
-----------------------------------------------------------

Review request for drill and Jacques Nadeau.
Repository: drill-git
Description
-------

1. Configuring Hive Metastore as storage engine in Drill
   add the metastore config to conf/storage-engines.json.
 Ex.
    "hive-derby" :
       {
         "type":"hive",
         "config" :
           {
             "hive.metastore.uris" : "localhost:9083",
             "hive.metastore.sasl.enabled" : "false"
           }
       }
   To embed the metastore rather than connecting to a remote server, set the "hive.metastore.uris" to empty and give metastore backend related properties in config section.
2. Implement schema interfaces to list tables in Hive metastore and covert hive types into Drill/Sql types.
Currently hive tables are mapped in two level schema. First level schema is used for the hive metastore (you can have more than one hive metastore configured at the same time). Second level of schema is the database within the metastore. See an example of the "show tables" below. In order to query a table "kv_seq" in database "testdb1" in hive metastore configured as schema "hive-derby", refer to the table as "dbname"."tablename". Ex. SELECT * FROM "testdb1"."kv_seq";

$ ./bin/sqlline -u "jdbc:drill:schema=hive-derby" -n admin -p admin
scan complete in 4062ms
Connecting to jdbc:drill:schema=hive-derby
Connected to: Drill (version 1.0)
Driver: Apache Drill JDBC Driver (version 1.0)
Autocommit status: true
Transaction isolation: TRANSACTION_REPEATABLE_READ
sqlline version ??? by Marc Prud'hommeaux
0: jdbc:drill:schema=hive-derby> !tables
+------------+--------------+--------------------+---------------+----------+-----------+-------------+---------+
| TABLE_CAT  | TABLE_SCHEM  |     TABLE_NAME     |  TABLE_TYPE   | REMARKS  | TYPE_CAT  | TYPE_SCHEM  | TYPE_NA |
+------------+--------------+--------------------+---------------+----------+-----------+-------------+---------+
| null       | hive-derby   | testdb1.kv_seq     | TABLE         | null     | null      | null        | null    |
| null       | hive-derby   | default.all_types  | TABLE         | null     | null      | null        | null    |
| null       | hive-derby   | default.kv_text    | TABLE         | null     | null      | null        | null    |
| null       | metadata     | COLUMNS            | SYSTEM_TABLE  | null     | null      | null        | null    |
| null       | metadata     | TABLES             | SYSTEM_TABLE  | null     | null      | null        | null    |
+------------+--------------+--------------------+---------------+----------+-----------+-------------+---------+
0: jdbc:drill:schema=hive-derby>
Diffs
-----

  contrib/pom.xml dc07979
  contrib/storage-hive/pom.xml PRE-CREATION
  distribution/src/assemble/bin.xml fdd6c70
  exec/java-exec/pom.xml 2ffdae9
  exec/java-exec/src/main/java/org/apache/drill/exec/store/hive/HiveReadEntry.java PRE-CREATION
  exec/java-exec/src/main/java/org/apache/drill/exec/store/hive/HiveSchemaProvider.java PRE-CREATION
  exec/java-exec/src/main/java/org/apache/drill/exec/store/hive/HiveStorageEngineConfig.java PRE-CREATION
  sqlparser/pom.xml 34622f7
  sqlparser/src/main/java/org/apache/drill/jdbc/DrillHandler.java 51600ec
  sqlparser/src/main/java/org/apache/drill/sql/client/full/HiveDatabaseSchema.java PRE-CREATION
  sqlparser/src/main/java/org/apache/drill/sql/client/full/HiveSchema.java PRE-CREATION
  sqlparser/src/test/java/org/apache/drill/jdbc/test/FullEngineTest.java f271bad
  sqlparser/src/test/resources/storage-engines.json PRE-CREATION

Diff: https://reviews.apache.org/r/17626/diff/
Testing
-------

Included an e2e test which lists the tables in Sample hive metastore. Sample derby metastore (sample-data/drill_hive_db) and warehouse (sample-data/drill_hive_wh) are not included in this patch. They can be found in this commit: https://github.com/vkorukanti/incubator-drill/commit/55265368b642e18700ad067a69a3a64f07280067
Thanks,

Venki Korukanti