Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Drill, mail # dev - Review Request: Adding JSONRecordReader


Copy link to this message
-
Re: Review Request: Adding JSONRecordReader
Timothy Chen 2013-06-04, 05:31


> On June 3, 2013, 4:01 p.m., Jacques Nadeau wrote:
> > sandbox/prototype/exec/java-exec/src/main/java/org/apache/drill/exec/server/DrillbitContext.java, line 54
> > <https://reviews.apache.org/r/11587/diff/1/?file=299723#file299723line54>
> >
> >     Why did you remove these?

There is already another identical checkNull for context two lines above
> On June 3, 2013, 4:01 p.m., Jacques Nadeau wrote:
> > sandbox/prototype/exec/java-exec/src/main/java/org/apache/drill/exec/store/JSONRecordReader.java, line 96
> > <https://reviews.apache.org/r/11587/diff/1/?file=299725#file299725line96>
> >
> >     Ideally, we would maintain non-changing vectors across batches rather than recreating each time.  This is fine for now, though

I see, I thought from our conversations you wanted the batches to recreate every batch.
I can easily do that though, if I just don't remove the mapping and start from index 0 again. Want me to do so?
> On June 3, 2013, 4:01 p.m., Jacques Nadeau wrote:
> > sandbox/prototype/exec/java-exec/src/main/java/org/apache/drill/exec/schema/Field.java, line 120
> > <https://reviews.apache.org/r/11587/diff/1/?file=299716#file299716line120>
> >
> >     DataMode should OPTIONAL for Nullable fields.

Ah, something I wanted to ask you too. Currently the TypeHelper getNewValueVector requires everything to be DataMode.REQUIRED, then I'll add the Nullable ones in the Optional switch then.
- Timothy
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/11587/#review21278
-----------------------------------------------------------
On May 31, 2013, 11:47 p.m., Timothy Chen wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/11587/
> -----------------------------------------------------------
>
> (Updated May 31, 2013, 11:47 p.m.)
>
>
> Review request for drill and Jacques Nadeau.
>
>
> Description
> -------
>
> Added the JSONRecordReader based on the previous ScanJson work.
>  Does not support nested fields, maps or lists yet.
>  Currently it detects to move on to the next batch when any of the field batch cannot hold another item for the current item being written. This also assumes the default batch size can always hold at least one item from any field (which only is a problem for variable length vectors).
>
>
> Diffs
> -----
>
>   sandbox/prototype/common/src/main/java/org/apache/drill/common/physical/schema/DiffSchema.java PRE-CREATION
>   sandbox/prototype/common/src/main/java/org/apache/drill/common/physical/schema/Field.java PRE-CREATION
>   sandbox/prototype/common/src/main/java/org/apache/drill/common/physical/schema/IdGenerator.java PRE-CREATION
>   sandbox/prototype/common/src/main/java/org/apache/drill/common/physical/schema/ListSchema.java PRE-CREATION
>   sandbox/prototype/common/src/main/java/org/apache/drill/common/physical/schema/NamedField.java PRE-CREATION
>   sandbox/prototype/common/src/main/java/org/apache/drill/common/physical/schema/ObjectSchema.java PRE-CREATION
>   sandbox/prototype/common/src/main/java/org/apache/drill/common/physical/schema/OrderedField.java PRE-CREATION
>   sandbox/prototype/common/src/main/java/org/apache/drill/common/physical/schema/RecordSchema.java PRE-CREATION
>   sandbox/prototype/common/src/main/java/org/apache/drill/common/physical/schema/SchemaIdGenerator.java PRE-CREATION
>   sandbox/prototype/common/src/main/java/org/apache/drill/common/physical/schema/json/jackson/JacksonHelper.java PRE-CREATION
>   sandbox/prototype/exec/java-exec/src/main/java/org/apache/drill/exec/record/DeadBuf.java dafb68c
>   sandbox/prototype/exec/java-exec/src/main/java/org/apache/drill/exec/record/vector/AbstractFixedValueVector.java b32f067
>   sandbox/prototype/exec/java-exec/src/main/java/org/apache/drill/exec/record/vector/BaseValueVector.java b001add
>   sandbox/prototype/exec/java-exec/src/main/java/org/apache/drill/exec/record/vector/NullableFixed4.java cc18538