Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hive, mail # dev - Review Request 16747: Add file pruning into Hive


Copy link to this message
-
Re: Review Request 16747: Add file pruning into Hive
Sergey Shelukhin 2014-01-13, 21:58


> On Jan. 10, 2014, 6:02 p.m., Sergey Shelukhin wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java, line 2868
> > <https://reviews.apache.org/r/16747/diff/1/?file=419383#file419383line2868>
> >
> >     why make it a hashset now? or should it have always been one
>
> Navis Ryu wrote:
>     I'm little confusing on this. Would it be not possible to have multiple paths for an alias? I think that kind of scenario is not supported by current hive. Reverting to list.

just checking... if it makes sense its ok
> On Jan. 10, 2014, 6:02 p.m., Sergey Shelukhin wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java, line 2921
> > <https://reviews.apache.org/r/16747/diff/1/?file=419383#file419383line2921>
> >
> >     nit: could return Collection from the method if it's not hard to change
>
> Navis Ryu wrote:
>     It's used by other code parts including TEZ. Would it be better to leave it as-is?

probably better to keep as is then... thanks
> On Jan. 10, 2014, 6:02 p.m., Sergey Shelukhin wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/plan/MapWork.java, line 559
> > <https://reviews.apache.org/r/16747/diff/1/?file=419395#file419395line559>
> >
> >     why is it recreating the list? maybe use addAll if it is needed?
>
> Navis Ryu wrote:
>     to convert String to Path?

ah, ic. Thanks
- Sergey
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/16747/#review31518
-----------------------------------------------------------
On Jan. 13, 2014, 4:33 a.m., Navis Ryu wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/16747/
> -----------------------------------------------------------
>
> (Updated Jan. 13, 2014, 4:33 a.m.)
>
>
> Review request for hive.
>
>
> Bugs: HIVE-1662
>     https://issues.apache.org/jira/browse/HIVE-1662
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> now hive support filename virtual column.
> if a file name filter presents in a query, hive should be able to only add files which passed the filter to input paths.
>
>
> Diffs
> -----
>
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 16d54c6
>   ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java 96a78fc
>   ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java fccea89
>   ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecDriver.java 5511bca
>   ql/src/java/org/apache/hadoop/hive/ql/exec/mr/MapRedTask.java a7e2253
>   ql/src/java/org/apache/hadoop/hive/ql/index/IndexPredicateAnalyzer.java e66c22c
>   ql/src/java/org/apache/hadoop/hive/ql/io/HiveFileFormatUtils.java 4be56f3
>   ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java 99172d4
>   ql/src/java/org/apache/hadoop/hive/ql/metadata/FilePrunningPredicateHandler.java PRE-CREATION
>   ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveStoragePredicateHandler.java 9f35575
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/AbstractJoinTaskDispatcher.java 33ef581
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/index/IndexWhereProcessor.java 5c6751c
>   ql/src/java/org/apache/hadoop/hive/ql/parse/MapReduceCompiler.java 76f5a31
>   ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeDescUtils.java 96c8d89
>   ql/src/java/org/apache/hadoop/hive/ql/plan/MapWork.java 9929275
>   ql/src/java/org/apache/hadoop/hive/ql/plan/MapredWork.java f3203bf
>   ql/src/java/org/apache/hadoop/hive/ql/plan/PlanUtils.java 6ee6bee
>   ql/src/java/org/apache/hadoop/hive/ql/plan/TableScanDesc.java 9c35890
>   ql/src/java/org/apache/hadoop/hive/ql/ppd/OpProcFactory.java 40298e1
>   ql/src/test/queries/clientpositive/file_pruning.q PRE-CREATION
>   ql/src/test/results/clientpositive/file_pruning.q.out PRE-CREATION
>
> Diff: https://reviews.apache.org/r/16747/diff/
>
>
> Testing
> -------
>