Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Hive >> mail # dev >> Re: Review Request 10857: Draft patch for review. Based on phase 1 mentioned in design doc.


Copy link to this message
-
Re: Review Request 10857: Draft patch for review. Based on phase 1 mentioned in design doc.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/10857/
-----------------------------------------------------------

(Updated Sept. 17, 2013, 11:56 p.m.)
Review request for hive.
Changes
-------

Updating patch. Addresses all comments captured so far form Brock, Allan and Eugene.
Also added unit tests.
Bugs: HIVE-4196
    https://issues.apache.org/jira/browse/HIVE-4196
Repository: hive-git
Description
-------

Draft patch for review. based on phase 1 mentioned in design doc. Deviates slighlty from doc in the follow ways...
1) adds a couple of (temporary) rest calls to enable/disable streaming on a table. Later these will be replaced with support in DDL.
2) Also also HTTP methods are GET for easy testing with web browser
3) Authentication disabled on the new streaming HTTP methods
Usage Examples on db named 'sdb' & table named 'log' :

1) Setup db & table with single partition column 'date':
hcat -e "create database sdb; use sdb; create table log(msg string, region string) partitioned by (date string) ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' LINES TERMINATED BY '\n' STORED AS TEXTFILE; "
2) To check streaming status:
http://localhost:50111/templeton/v1/streaming/status?database=sdb&table=log
3) Enable Streaming:
http://localhost:50111/templeton/v1/streaming/enable?database=sdb&table=log&col=date&value=1000
4) Get Chunk File to write to:
http://localhost:50111/templeton/v1/streaming/chunkget?database=sdb&table=log&schema=blah&format=blah&record_separator=blah&field_separator=blah
5) Commit Chunk File:
http://localhost:50111/templeton/v1/streaming/chunkcommit?database=sdb&table=log&chunkfile=/user/hive/streaming/tmp/sdb/log/2
6) Abort Chunk File:
http://localhost:50111/templeton/v1/streaming/chunkabort?database=sdb&table=log&chunkfile=/user/hive/streaming/tmp/sdb/log/3
7) Roll Partition:
http://localhost:50111/templeton/v1/streaming/partitionroll?database=sdb&table=log&partition_column=date&partition_value=3000
Diffs (updated)
-----

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1af68a6
  hcatalog/webhcat/svr/pom.xml 16a870e
  hcatalog/webhcat/svr/src/main/java/org/apache/hive/hcatalog/templeton/HcatStreamingDelegator.java PRE-CREATION
  hcatalog/webhcat/svr/src/main/java/org/apache/hive/hcatalog/templeton/Main.java 0f37278
  hcatalog/webhcat/svr/src/main/java/org/apache/hive/hcatalog/templeton/Server.java d8b6ce2
  hcatalog/webhcat/svr/src/test/java/org/apache/hive/hcatalog/templeton/TestStreaming.java PRE-CREATION
  hcatalog/webhcat/svr/src/test/java/org/apache/hive/hcatalog/templeton/tool/HiveJdbcClient.java PRE-CREATION
  hcatalog/webhcat/svr/src/test/java/org/apache/hive/hcatalog/templeton/tool/WebHCatClient.java PRE-CREATION
  metastore/if/hive_metastore.thrift 3fea057
  metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 39dda92
  metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java e165b2d
  metastore/src/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java a2d6b1b
  metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java a27243d
  metastore/src/java/org/apache/hadoop/hive/metastore/RawStore.java e410c3a
  metastore/src/model/org/apache/hadoop/hive/metastore/model/MTable.java 2a78ce9
  metastore/src/model/package.jdo c42b5b0
  metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java 8066784
  metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java 0f9b16c
  metastore/src/test/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java 80fbbac

Diff: https://reviews.apache.org/r/10857/diff/
Testing
-------

Manual testing only
Thanks,

Roshan Naik

+
Roshan Naik 2013-09-17, 23:58