Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hive, mail # dev - Review Request 14221: HIVE-4113: Optimize select count(1) with RCFile and Orc


Copy link to this message
-
Re: Review Request 14221: HIVE-4113: Optimize select count(1) with RCFile and Orc
Yin Huai 2013-09-21, 01:46

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14221/
-----------------------------------------------------------

(Updated Sept. 21, 2013, 1:45 a.m.)
Review request for hive.
Bugs: HIVE-4113
    https://issues.apache.org/jira/browse/HIVE-4113
Repository: hive-git
Description
-------

Modifies ColumnProjectionUtils such there are two flags. One for the column ids and one indicating whether all columns should be read. Additionally the patch updates all locations which uses the old method of empty string indicating all columns should be read.

The automatic formatter generated by ant eclipse-files is fairly aggressive so there are some unrelated import/whitespace cleanup.

This one is based on https://reviews.apache.org/r/11770/ and has been rebased to the latest trunk.
Diffs (updated)
-----

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 9f37d0c
  conf/hive-default.xml.template 545026d
  hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableInputFormat.java 766056b
  hcatalog/core/src/main/java/org/apache/hive/hcatalog/mapreduce/HCatBaseInputFormat.java 553446a
  hcatalog/core/src/test/java/org/apache/hive/hcatalog/mapreduce/TestHCatPartitioned.java 577e06d
  hcatalog/hcatalog-pig-adapter/src/test/java/org/apache/hive/hcatalog/pig/TestHCatLoader.java d38bb8d
  ql/src/java/org/apache/hadoop/hive/ql/Driver.java 31a52ba
  ql/src/java/org/apache/hadoop/hive/ql/exec/FetchTask.java df2ccf1
  ql/src/java/org/apache/hadoop/hive/ql/exec/SMBMapJoinOperator.java ab0494e
  ql/src/java/org/apache/hadoop/hive/ql/exec/TableScanOperator.java a5a8943
  ql/src/java/org/apache/hadoop/hive/ql/exec/mr/MapredLocalTask.java 0f29a0e
  ql/src/java/org/apache/hadoop/hive/ql/io/BucketizedHiveInputFormat.java 49145b7
  ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java cccdc1b
  ql/src/java/org/apache/hadoop/hive/ql/io/RCFile.java a83f223
  ql/src/java/org/apache/hadoop/hive/ql/io/RCFileRecordReader.java 9521060
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java 50c5093
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/ColumnPrunerProcFactory.java ed14e82
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMRFileSink1.java 1ede6d7
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMRUnion1.java 52e9e6b
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java 2259977
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/Optimizer.java b97d869
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/MetadataOnlyOptimizer.java 0550bf6
  ql/src/test/org/apache/hadoop/hive/ql/io/PerformTestRCFileAndSeqFile.java fb9fca1
  ql/src/test/org/apache/hadoop/hive/ql/io/TestRCFile.java dd1276d
  ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestInputOutputFormat.java 83c5c38
  ql/src/test/queries/clientpositive/binary_table_colserde.q eadf07d
  ql/src/test/results/clientpositive/auto_join0.q.out a75c01c
  ql/src/test/results/clientpositive/auto_join15.q.out 6fb0ea6
  ql/src/test/results/clientpositive/auto_join18.q.out 945af67
  ql/src/test/results/clientpositive/auto_join18_multi_distinct.q.out 500df42
  ql/src/test/results/clientpositive/auto_join20.q.out 6dd8ff7
  ql/src/test/results/clientpositive/auto_join27.q.out aac778c
  ql/src/test/results/clientpositive/binary_table_colserde.q.out 69a6c6e
  ql/src/test/results/clientpositive/correlationoptimizer1.q.out 2b8a149
  ql/src/test/results/clientpositive/correlationoptimizer10.q.out 5886d4c
  ql/src/test/results/clientpositive/correlationoptimizer11.q.out 72439f8
  ql/src/test/results/clientpositive/correlationoptimizer12.q.out 22dfce9
  ql/src/test/results/clientpositive/correlationoptimizer13.q.out 5eed255
  ql/src/test/results/clientpositive/correlationoptimizer14.q.out 99263aa
  ql/src/test/results/clientpositive/correlationoptimizer15.q.out 76318b5
  ql/src/test/results/clientpositive/correlationoptimizer2.q.out 6b97a78
  ql/src/test/results/clientpositive/correlationoptimizer3.q.out a6691ae
  ql/src/test/results/clientpositive/correlationoptimizer4.q.out 3605619
  ql/src/test/results/clientpositive/correlationoptimizer5.q.out 17692f5
  ql/src/test/results/clientpositive/correlationoptimizer6.q.out d8e1f29
  ql/src/test/results/clientpositive/correlationoptimizer7.q.out 7a17edf
  ql/src/test/results/clientpositive/correlationoptimizer8.q.out 2a60d81
  ql/src/test/results/clientpositive/correlationoptimizer9.q.out 4a8fea8
  ql/src/test/results/clientpositive/ctas_hadoop20.q.out 48bf035
  serde/src/java/org/apache/hadoop/hive/serde2/ColumnProjectionUtils.java 0b3ef7b
  serde/src/java/org/apache/hadoop/hive/serde2/columnar/ColumnarSerDe.java 11f5f07
  serde/src/java/org/apache/hadoop/hive/serde2/columnar/ColumnarStruct.java 1335446
  serde/src/java/org/apache/hadoop/hive/serde2/columnar/ColumnarStructBase.java e1270cc
  serde/src/java/org/apache/hadoop/hive/serde2/columnar/LazyBinaryColumnarSerDe.java b717278
  serde/src/java/org/apache/hadoop/hive/serde2/columnar/LazyBinaryColumnarStruct.java 0317024
  serde/src/test/org/apache/hadoop/hive/serde2/TestColumnProjectionUtils.java PRE-CREATION

Diff: https://reviews.apache.org/r/14221/diff/
Testing
Thanks,

Yin Huai