Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Hive, mail # dev - Review Request 14243: HIVE-5325: Implement statistics providing ORC writer and reader interfaces


+
j.prasanth.j@... 2013-09-20, 01:24
Copy link to this message
-
Re: Review Request 14243: HIVE-5325: Implement statistics providing ORC writer and reader interfaces
j.prasanth.j@... 2013-09-24, 22:18

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14243/
-----------------------------------------------------------

(Updated Sept. 24, 2013, 10:18 p.m.)
Review request for hive, Ashutosh Chauhan and Owen O'Malley.
Changes
-------

Refreshed the patch after HIVE-5324 changes.
Bugs: HIVE-5325
    https://issues.apache.org/jira/browse/HIVE-5325
Repository: hive-git
Description
-------

HIVE-5324 adds new interfaces that can be implemented by ORC reader/writer to provide statistics. Writer provided statistics is used to update table/partition level statistics in metastore. Reader provided statistics can be used for reducer estimation, CBO etc. in the absence of metastore statistics.
Diffs (updated)
-----

  ql/src/java/org/apache/hadoop/hive/ql/io/orc/BinaryColumnStatistics.java PRE-CREATION
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/ColumnStatisticsImpl.java 6268617
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcOutputFormat.java c80fb02
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/ReaderImpl.java c454f32
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/StringColumnStatistics.java 72e779a
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/WriterImpl.java 44961ce
  ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUtils.java PRE-CREATION
  ql/src/protobuf/org/apache/hadoop/hive/ql/io/orc/orc_proto.proto edbf822
  ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestInputOutputFormat.java 34b2305
  ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestOrcFile.java e6569f4
  ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestOrcNullOptimization.java b93db84
  ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestOrcSerDeStats.java PRE-CREATION
  ql/src/test/resources/orc-file-dump-dictionary-threshold.out 003c132
  ql/src/test/resources/orc-file-dump.out fac5326

Diff: https://reviews.apache.org/r/14243/diff/
Testing
-------

ORC related unit and qfile tests are passing.
Thanks,

Prasanth_J

+
Ashutosh Chauhan 2013-09-30, 16:16
+
j.prasanth.j@... 2013-10-01, 01:12
+
j.prasanth.j@... 2013-10-01, 01:13
+
j.prasanth.j@... 2013-10-01, 01:55