Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Hive >> mail # dev >> Review Request: Draft patch for review. Based on phase 1 mentioned in design doc.


Copy link to this message
-
Review Request: Draft patch for review. Based on phase 1 mentioned in design doc.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/10857/
-----------------------------------------------------------

Review request for hive.
Description
-------

Draft patch for review. based on phase 1 mentioned in design doc. Deviates slighlty from doc in the follow ways...
1) adds a couple of (temporary) rest calls to enable/disable streaming on a table. Later these will be replaced with support in DDL.
2) Also also HTTP methods are GET for easy testing with web browser
3) Authentication disabled on the new streaming HTTP methods
Usage Examples on db named 'sdb' & table named 'log' :

1) Setup db & table with single partition column 'date':
hcat -e "create database sdb; use sdb; create table log(msg string, region string) partitioned by (date string) ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' LINES TERMINATED BY '\n' STORED AS TEXTFILE; "
2) To check streaming status:
http://localhost:50111/templeton/v1/streaming/status?database=sdb&table=log
3) Enable Streaming:
http://localhost:50111/templeton/v1/streaming/enable?database=sdb&table=log&col=date&value=1000
4) Get Chunk File to write to:
http://localhost:50111/templeton/v1/streaming/chunkget?database=sdb&table=log&schema=blah&format=blah&record_separator=blah&field_separator=blah
5) Commit Chunk File:
http://localhost:50111/templeton/v1/streaming/chunkcommit?database=sdb&table=log&chunkfile=/user/hive/streaming/tmp/sdb/log/2
6) Abort Chunk File:
http://localhost:50111/templeton/v1/streaming/chunkabort?database=sdb&table=log&chunkfile=/user/hive/streaming/tmp/sdb/log/3
7) Roll Partition:
http://localhost:50111/templeton/v1/streaming/partitionroll?database=sdb&table=log&partition_column=date&partition_value=3000
This addresses bug HIVE-4196.
    https://issues.apache.org/jira/browse/HIVE-4196
Diffs
-----

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java c61d95b
  hcatalog/webhcat/svr/src/main/java/org/apache/hcatalog/templeton/HcatStreamingDelegator.java PRE-CREATION
  hcatalog/webhcat/svr/src/main/java/org/apache/hcatalog/templeton/Server.java 29ac4b3
  metastore/if/hive_metastore.thrift c2051f4
  metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore.h 7b31d28
  metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore.cpp 3d69472
  metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore_server.skeleton.cpp 3b90b44
  metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/SkewedInfo.java d8d6e71
  metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/SkewedValueList.java 030b54a
  metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ThriftHiveMetastore.java 5929cda
  metastore/src/gen/thrift/gen-php/metastore/ThriftHiveMetastore.php a69d214
  metastore/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore-remote 6fd2cce
  metastore/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore.py 9b856e5
  metastore/src/gen/thrift/gen-rb/thrift_hive_metastore.rb 25aa30c
  metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java dc14084
  metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java cef50f4
  metastore/src/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java a2d6b1b
  metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 2079337
  metastore/src/java/org/apache/hadoop/hive/metastore/RawStore.java 233fb46
  metastore/src/model/org/apache/hadoop/hive/metastore/model/MTable.java 2a78ce9
  metastore/src/model/package.jdo a84d2bf
  metastore/src/test/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java 00eb0b4

Diff: https://reviews.apache.org/r/10857/diff/
Testing
-------

Manual testing only
Thanks,

Roshan Naik