Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Hive, mail # dev - Re: Review Request: HIVE-2206: add a new optimizer for query correlation discovery and optimization


+
Yin Huai 2012-11-03, 01:50
+
Yin Huai 2012-11-12, 22:21
Copy link to this message
-
Re: Review Request: HIVE-2206: add a new optimizer for query correlation discovery and optimization
Yin Huai 2012-11-19, 19:51

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/7126/
-----------------------------------------------------------

(Updated Nov. 19, 2012, 7:51 p.m.)
Review request for hive.
Changes
-------

Correlation optimizer will guess which join operators at the bottom (input tables are not intermediate tables) will be optimized by auto join convert and ignore those join operators in the optimization of correlation optimizer.
Description
-------

This optimizer exploits intra-query correlations and merges multiple correlated MapReduce jobs into one jobs. Open a new request since I have been working on hive-git.
This addresses bug HIVE-2206.
    https://issues.apache.org/jira/browse/HIVE-2206
Diffs (updated)
-----

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 9fa9525
  conf/hive-default.xml.template f332f3a
  ql/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/ql/plan/api/OperatorType.java 7c4c413
  ql/src/java/org/apache/hadoop/hive/ql/exec/BaseReduceSinkOperator.java PRE-CREATION
  ql/src/java/org/apache/hadoop/hive/ql/exec/CorrelationCompositeOperator.java PRE-CREATION
  ql/src/java/org/apache/hadoop/hive/ql/exec/CorrelationLocalSimulativeReduceSinkOperator.java PRE-CREATION
  ql/src/java/org/apache/hadoop/hive/ql/exec/CorrelationReducerDispatchOperator.java PRE-CREATION
  ql/src/java/org/apache/hadoop/hive/ql/exec/ExecReducer.java 18a9bd2
  ql/src/java/org/apache/hadoop/hive/ql/exec/GroupByOperator.java 46daeb2
  ql/src/java/org/apache/hadoop/hive/ql/exec/Operator.java 68302f8
  ql/src/java/org/apache/hadoop/hive/ql/exec/OperatorFactory.java 0c22141
  ql/src/java/org/apache/hadoop/hive/ql/exec/ReduceSinkOperator.java 919a140
  ql/src/java/org/apache/hadoop/hive/ql/exec/TableScanOperator.java 1469325
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/CorrelationOptimizer.java PRE-CREATION
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/CorrelationOptimizerUtils.java PRE-CREATION
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java edde378
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/Optimizer.java d1555e2
  ql/src/java/org/apache/hadoop/hive/ql/parse/ParseContext.java 2bf284d
  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 330aa52
  ql/src/java/org/apache/hadoop/hive/ql/plan/BaseReduceSinkDesc.java PRE-CREATION
  ql/src/java/org/apache/hadoop/hive/ql/plan/CorrelationCompositeDesc.java PRE-CREATION
  ql/src/java/org/apache/hadoop/hive/ql/plan/CorrelationLocalSimulativeReduceSinkDesc.java PRE-CREATION
  ql/src/java/org/apache/hadoop/hive/ql/plan/CorrelationReducerDispatchDesc.java PRE-CREATION
  ql/src/java/org/apache/hadoop/hive/ql/plan/MapredWork.java 5a9f064
  ql/src/java/org/apache/hadoop/hive/ql/plan/ReduceSinkDesc.java b33d616
  ql/src/java/org/apache/hadoop/hive/ql/plan/TableScanDesc.java 9a95efd
  ql/src/test/org/apache/hadoop/hive/ql/exec/TestExecDriver.java 6f8bc47
  ql/src/test/queries/clientpositive/correlationoptimizer1.q PRE-CREATION
  ql/src/test/queries/clientpositive/correlationoptimizer2.q PRE-CREATION
  ql/src/test/queries/clientpositive/correlationoptimizer3.q PRE-CREATION
  ql/src/test/queries/clientpositive/correlationoptimizer4.q PRE-CREATION
  ql/src/test/queries/clientpositive/correlationoptimizer5.q PRE-CREATION
  ql/src/test/results/clientpositive/correlationoptimizer1.q.out PRE-CREATION
  ql/src/test/results/clientpositive/correlationoptimizer2.q.out PRE-CREATION
  ql/src/test/results/clientpositive/correlationoptimizer3.q.out PRE-CREATION
  ql/src/test/results/clientpositive/correlationoptimizer4.q.out PRE-CREATION
  ql/src/test/results/clientpositive/correlationoptimizer5.q.out PRE-CREATION
  ql/src/test/results/compiler/plan/groupby1.q.xml cd0d6e4
  ql/src/test/results/compiler/plan/groupby2.q.xml 7b07f02
  ql/src/test/results/compiler/plan/groupby3.q.xml a6a1986
  ql/src/test/results/compiler/plan/groupby5.q.xml 25e3583

Diff: https://reviews.apache.org/r/7126/diff/
Testing

All tests pass.
Thanks,

Yin Huai
+
Ashutosh Chauhan 2013-01-13, 01:00