Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hive, mail # dev - Review Request 16076: Fix HS2 ResultSet Serialization Performance Regression


Copy link to this message
-
Re: Review Request 16076: Fix HS2 ResultSet Serialization Performance Regression
Carl Steinbach 2013-12-16, 05:37

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/16076/#review30434
-----------------------------------------------------------

service/if/TCLIService.thrift
<https://reviews.apache.org/r/16076/#comment58226>

    Need to add HIVE_CLI_SERVICE_PROTOCOL_V5 and update existing references to HIVE_CLI_SERVICE_PROTOCOL_V5.

service/if/TCLIService.thrift
<https://reviews.apache.org/r/16076/#comment58229>

    Please add TBoolColumn, TByteColumn, etc instead of redefining the existing T*Value structs.

service/if/TCLIService.thrift
<https://reviews.apache.org/r/16076/#comment58228>

    These changes break compatibility with older HiveServer2 clients. Instead I think we want to make it possible for a client to pick between the existing serialization format and the new column-oriented serialization format by setting a new optional field in TFetchResultsReq.

service/if/TCLIService.thrift
<https://reviews.apache.org/r/16076/#comment58230>

    We aren't using TColumn right now so it should be ok to redefine the contents of this struct. Also, it may be worth trying to save a bit of space by moving the "binary nulls" field outside of the union of the individual T*Column structs, e.g:
    
    union TColumn {
      1: TBoolColumn boolColumn
      ...
      7: TStringColumn stringColumn
    }
    
    struct TNullableColumn {
      1: TColumn column
      2: binary nulls
    }

service/if/TCLIService.thrift
<https://reviews.apache.org/r/16076/#comment58233>

    Please add a new struct TColumnSet instead of modifying TRowSet.

service/if/TCLIService.thrift
<https://reviews.apache.org/r/16076/#comment58232>

    please add:
    
    4: optional TResultSetType tResultSetType
    
    where TResultSetType is an enum that defaults to ROW_ORIENTED_RESULTSET

service/if/TCLIService.thrift
<https://reviews.apache.org/r/16076/#comment58231>

    please add:
    
    4: optional TColumnSet columnResults
    
- Carl Steinbach
On Dec. 16, 2013, 2:04 a.m., Navis Ryu wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/16076/
> -----------------------------------------------------------
>
> (Updated Dec. 16, 2013, 2:04 a.m.)
>
>
> Review request for hive.
>
>
> Bugs: HIVE-3746
>     https://issues.apache.org/jira/browse/HIVE-3746
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> serialize result set in columnar format
>
>
> Diffs
> -----
>
>   itests/hive-unit/src/test/java/org/apache/hive/jdbc/miniHS2/TestHiveServer2.java eb08628
>   jdbc/src/java/org/apache/hive/jdbc/HiveBaseResultSet.java b02f374
>   jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java 061337d
>   service/if/TCLIService.thrift 62a9730
>   service/src/gen/thrift/gen-cpp/TCLIService_types.h 853bb4c
>   service/src/gen/thrift/gen-cpp/TCLIService_types.cpp 7ab1310
>   service/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/service/ThriftHive.java 1c44789
>   service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TBinaryValue.java PRE-CREATION
>   service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TBoolValue.java c7495ee
>   service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TByteValue.java 23d9693
>   service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TColumn.java 497cc01
>   service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TColumnValue.java 44da2cd
>   service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TDoubleValue.java d215736
>   service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TExecuteStatementReq.java ea656ac
>   service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TGetTablesReq.java 1cb5147
>   service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TI16Value.java bb5ae96