Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Drill >> mail # dev >> Review Request: Adding JSONRecordReader


Copy link to this message
-
Re: Review Request: Adding JSONRecordReader

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/11587/
-----------------------------------------------------------

(Updated June 6, 2013, 8:19 p.m.)
Review request for drill and Jacques Nadeau.
Changes
-------

Reallocate vector every batch
Description
-------

Added the JSONRecordReader based on the previous ScanJson work.
 Does not support nested fields, maps or lists yet.
 Currently it detects to move on to the next batch when any of the field batch cannot hold another item for the current item being written. This also assumes the default batch size can always hold at least one item from any field (which only is a problem for variable length vectors).
Diffs (updated)
-----

  sandbox/prototype/common/src/main/java/org/apache/drill/common/logical/data/Scan.java c8d396b
  sandbox/prototype/exec/java-exec/src/main/java/org/apache/drill/exec/record/DeadBuf.java dafb68c
  sandbox/prototype/exec/java-exec/src/main/java/org/apache/drill/exec/record/vector/AbstractFixedValueVector.java b32f067
  sandbox/prototype/exec/java-exec/src/main/java/org/apache/drill/exec/record/vector/BaseValueVector.java b001add
  sandbox/prototype/exec/java-exec/src/main/java/org/apache/drill/exec/record/vector/Fixed4.java 650029d
  sandbox/prototype/exec/java-exec/src/main/java/org/apache/drill/exec/record/vector/NullableFixed4.java cc18538
  sandbox/prototype/exec/java-exec/src/main/java/org/apache/drill/exec/record/vector/NullableValueVector.java 692ab87
  sandbox/prototype/exec/java-exec/src/main/java/org/apache/drill/exec/record/vector/NullableVarLen4.java PRE-CREATION
  sandbox/prototype/exec/java-exec/src/main/java/org/apache/drill/exec/record/vector/TypeHelper.java 8e89c41
  sandbox/prototype/exec/java-exec/src/main/java/org/apache/drill/exec/record/vector/VarLen1.java d87029d
  sandbox/prototype/exec/java-exec/src/main/java/org/apache/drill/exec/record/vector/VarLen2.java ebd440a
  sandbox/prototype/exec/java-exec/src/main/java/org/apache/drill/exec/record/vector/VarLen4.java b3cd712
  sandbox/prototype/exec/java-exec/src/main/java/org/apache/drill/exec/record/vector/VariableVector.java 4247f14
  sandbox/prototype/exec/java-exec/src/main/java/org/apache/drill/exec/rpc/InboundRpcMessage.java be1ff6b
  sandbox/prototype/exec/java-exec/src/main/java/org/apache/drill/exec/schema/DiffSchema.java 016e097
  sandbox/prototype/exec/java-exec/src/main/java/org/apache/drill/exec/schema/Field.java e19c099
  sandbox/prototype/exec/java-exec/src/main/java/org/apache/drill/exec/schema/IdGenerator.java 728e8e1
  sandbox/prototype/exec/java-exec/src/main/java/org/apache/drill/exec/schema/ListSchema.java efdc8fd
  sandbox/prototype/exec/java-exec/src/main/java/org/apache/drill/exec/schema/NamedField.java aa0d6aa
  sandbox/prototype/exec/java-exec/src/main/java/org/apache/drill/exec/schema/OrderedField.java 67fd2fa
  sandbox/prototype/exec/java-exec/src/main/java/org/apache/drill/exec/schema/SchemaRecorder.java 54a4e0e
  sandbox/prototype/exec/java-exec/src/main/java/org/apache/drill/exec/schema/json/jackson/JacksonHelper.java 0643710
  sandbox/prototype/exec/java-exec/src/main/java/org/apache/drill/exec/schema/json/jackson/PhysicalOperator.java e450ee9
  sandbox/prototype/exec/java-exec/src/main/java/org/apache/drill/exec/schema/json/jackson/PhysicalOperatorIterator.java bf4053e
  sandbox/prototype/exec/java-exec/src/main/java/org/apache/drill/exec/schema/json/jackson/ScanJson.java a1c30e9
  sandbox/prototype/exec/java-exec/src/main/java/org/apache/drill/exec/schema/transform/ProtobufSchemaTransformer.java a81a9d9
  sandbox/prototype/exec/java-exec/src/main/java/org/apache/drill/exec/schema/transform/SchemaTransformer.java 54d851f
  sandbox/prototype/exec/java-exec/src/main/java/org/apache/drill/exec/server/DrillbitContext.java d5aaab2
  sandbox/prototype/exec/java-exec/src/main/java/org/apache/drill/exec/store/BatchExceededException.java PRE-CREATION
  sandbox/prototype/exec/java-exec/src/main/java/org/apache/drill/exec/store/JSONRecordReader.java PRE-CREATION
  sandbox/prototype/exec/java-exec/src/main/java/org/apache/drill/exec/store/RecordReader.java 67c84ed
  sandbox/prototype/exec/java-exec/src/main/java/org/apache/drill/exec/store/VectorHolder.java PRE-CREATION
  sandbox/prototype/exec/java-exec/src/test/java/org/apache/drill/exec/store/JSONRecordReaderTest.java PRE-CREATION
  sandbox/prototype/exec/java-exec/src/test/resources/scan_json_test_1.json PRE-CREATION
  sandbox/prototype/exec/java-exec/src/test/resources/scan_json_test_2.json PRE-CREATION
  sandbox/prototype/exec/java-exec/src/test/resources/scan_json_test_3.json PRE-CREATION
  sandbox/prototype/exec/java-exec/src/test/resources/scan_json_test_4.json PRE-CREATION
  sandbox/prototype/exec/java-exec/src/test/resources/scan_json_test_5.json PRE-CREATION
  sandbox/prototype/exec/java-exec/src/test/resources/scan_json_test_6.json PRE-CREATION
  sandbox/prototype/exec/ref/src/main/java/org/apache/drill/exec/ref/rops/MultiLevelMap.java PRE-CREATION
  sandbox/prototype/exec/ref/src/main/java/org/apache/drill/exec/ref/rops/WindowingROP.java PRE-CREATION
  sandbox/prototype/exec/ref/src/test/resources/donuts.json 2d98b59
  sandbox/prototype/exec/ref/src/test/resources/simple_plan.json 0f38476
  sandbox/prototype/pom.xml 25f156d

Diff: https://reviews.apache.org/r/11587/diff/
Testing
Thanks,

Timothy Chen