Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hive >> mail # dev >> Review Request 12480: HIVE-4732 Reduce or eliminate the expensive Schema equals() check for AvroSerde


Copy link to this message
-
Re: Review Request 12480: HIVE-4732 Reduce or eliminate the expensive Schema equals() check for AvroSerde

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/12480/
-----------------------------------------------------------

(Updated Aug. 7, 2013, 2:13 a.m.)
Review request for hive, Ashutosh Chauhan and Jakob Homan.
Changes
-------

Add logic to avoid excessive logging for each record.
Bugs: HIVE-4732
    https://issues.apache.org/jira/browse/HIVE-4732
Repository: hive-git
Description
-------

>From our performance analysis, we found AvroSerde's schema.equals() call consumed a substantial amount ( nearly 40%) of time. This patch intends to minimize the number schema.equals() calls by pushing the check as late/fewer as possible.

At first, we added a unique id for each record reader which is then included in every AvroGenericRecordWritable. Then, we introduce two new data structures (one hashset and one hashmap) to store intermediate data to avoid duplicates checkings. Hashset contains all the record readers' IDs that don't need any re-encoding. On the other hand, HashMap contains the already used re-encoders. It works as cache and allows re-encoders reuse. With this change, our test shows nearly 40% reduction in Avro record reading time.
 
  
Diffs (updated)
-----

  ql/src/java/org/apache/hadoop/hive/ql/io/avro/AvroGenericRecordReader.java ed2a9af
  serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroDeserializer.java e994411
  serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroGenericRecordWritable.java 66f0348
  serde/src/test/org/apache/hadoop/hive/serde2/avro/TestAvroDeserializer.java 3828940
  serde/src/test/org/apache/hadoop/hive/serde2/avro/TestSchemaReEncoder.java 9af751b
  serde/src/test/org/apache/hadoop/hive/serde2/avro/Utils.java 2b948eb

Diff: https://reviews.apache.org/r/12480/diff/
Testing
-------
Thanks,

Mohammad Islam