Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hive, mail # dev - Review Request 15561: HIVE-5771 Constant propagation optimizer for Hive


Copy link to this message
-
Re: Review Request 15561: HIVE-5771 Constant propagation optimizer for Hive
Ted Xu 2014-01-18, 14:54


> On Nov. 19, 2013, 6:45 p.m., Xuefu Zhang wrote:
> > http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/ColumnPrunerProcFactory.java, line 398
> > <https://reviews.apache.org/r/15561/diff/2/?file=386971#file386971line398>
> >
> >     What is this fix for?

Because ColumnPruner is applied after constant propagation optimizer(ConstantPO), some expressions are already eliminated by ConstantPO. This statement here is to prevent ArrayOutOfBoundException in that cases.
- Ted
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/15561/#review29119
-----------------------------------------------------------
On Jan. 18, 2014, 2:54 p.m., Ted Xu wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/15561/
> -----------------------------------------------------------
>
> (Updated Jan. 18, 2014, 2:54 p.m.)
>
>
> Review request for hive.
>
>
> Bugs: HIVE-5771
>     https://issues.apache.org/jira/browse/HIVE-5771
>
>
> Repository: hive
>
>
> Description
> -------
>
> Currently there is no constant folding/propagation optimizer, all expressions are evaluated at runtime.
>
> HIVE-2470 did a great job on evaluating constants on UDF initializing phase, however, it is still a runtime evaluation and it doesn't propagate constants from a subquery to outside.
>
> It may reduce I/O and accelerate process if we introduce such an optimizer.
>
>
> Diffs
> -----
>
>   http://svn.apache.org/repos/asf/hive/trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1559345
>   http://svn.apache.org/repos/asf/hive/trunk/contrib/src/test/results/clientpositive/udf_example_add.q.out 1559345
>   http://svn.apache.org/repos/asf/hive/trunk/contrib/src/test/results/clientpositive/udf_example_format.q.out 1559345
>   http://svn.apache.org/repos/asf/hive/trunk/hbase-handler/src/test/results/positive/hbase_ppd_key_range.q.out 1559345
>   http://svn.apache.org/repos/asf/hive/trunk/hbase-handler/src/test/results/positive/hbase_pushdown.q.out 1559345
>   http://svn.apache.org/repos/asf/hive/trunk/hbase-handler/src/test/results/positive/ppd_key_ranges.q.out 1559345
>   http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/ColumnInfo.java 1559345
>   http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/ColumnPrunerProcFactory.java 1559345
>   http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/ConstantPropagate.java PRE-CREATION
>   http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/ConstantPropagateProcCtx.java PRE-CREATION
>   http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/ConstantPropagateProcFactory.java PRE-CREATION
>   http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/Optimizer.java 1559345
>   http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/queries/clientpositive/alter_partition_coltype.q 1559345
>   http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/queries/clientpositive/annotate_stats_filter.q 1559345
>   http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/queries/clientpositive/annotate_stats_part.q 1559345
>   http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/queries/clientpositive/annotate_stats_select.q 1559345
>   http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/queries/clientpositive/constprog1.q PRE-CREATION
>   http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/queries/clientpositive/constprog2.q PRE-CREATION
>   http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/queries/clientpositive/constprog_dp.q PRE-CREATION
>   http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/queries/clientpositive/constprog_type.q PRE-CREATION