Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig, mail # dev - Review Request: Add BigInteger and BigDecimal to Pig


Copy link to this message
-
Re: Review Request: Add BigInteger and BigDecimal to Pig
Jonathan Coveney 2013-01-22, 19:31


> On Jan. 19, 2013, 7:29 p.m., Mathias Herberts wrote:
> > src/org/apache/pig/data/DefaultTuple.java, line 359
> > <https://reviews.apache.org/r/9012/diff/1-2/?file=249804#file249804line359>
> >
> >     Since BigDecimal have a scale which is an int, nothing prevents a BigDecimal to have a scale which won't fit on a short, and thus whose string representation might also be longer than a short.
> >    
> >     Is I understand this code correctly (thanks to your explanation), if type if CHARARRAY, field length is encoded on a short? This will be troublesome if scale > 0x7ffff.

This is a bit annoying/confusing, but there are two internal "CHARARRAYS," either a CHARARRAY or a BIGCHARRAY. So if it's bigger than a short, it will be an int (ie a BIGCHARARRAY) and it should deserialized correctly. Literally all DataType.BIGINTEGER in this case signals is "what comes after is a string," which we then serialized and deserialize in accordance with the rest of pig.
- Jonathan
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/9012/#review15522
-----------------------------------------------------------
On Jan. 18, 2013, 10:11 p.m., Jonathan Coveney wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/9012/
> -----------------------------------------------------------
>
> (Updated Jan. 18, 2013, 10:11 p.m.)
>
>
> Review request for pig, Alan Gates and Mathias Herberts.
>
>
> Description
> -------
>
> This patch adds big integer and big decimal support to Pig. It could use more tests, something I'd appreciate feedback on (but I wanted to make sure the core implementation is good)
>
>
> This addresses bug PIG-2764.
>     https://issues.apache.org/jira/browse/PIG-2764
>
>
> Diffs
> -----
>
>   .gitignore cc62d7d
>   src/org/apache/pig/LoadCaster.java 574769b
>   src/org/apache/pig/PigWarning.java 5de075f
>   src/org/apache/pig/StoreCaster.java 5fe48de
>   src/org/apache/pig/backend/hadoop/BigDecimalWritable.java PRE-CREATION
>   src/org/apache/pig/backend/hadoop/BigIntegerWritable.java PRE-CREATION
>   src/org/apache/pig/backend/hadoop/HDataType.java 84a56b8
>   src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/JobControlCompiler.java 96fba6b
>   src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigBigDecimalRawComparator.java PRE-CREATION
>   src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigBigIntegerRawComparator.java PRE-CREATION
>   src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/partitioners/WeightedRangePartitioner.java 9749339
>   src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/PhysicalOperator.java f40eb43
>   src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/expressionOperators/Add.java c84b767
>   src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/expressionOperators/ConstantExpression.java db3840f
>   src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/expressionOperators/Divide.java 4656c28
>   src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/expressionOperators/EqualToExpr.java 6683beb
>   src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/expressionOperators/ExpressionOperator.java 2806336
>   src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/expressionOperators/GTOrEqualToExpr.java d64a080
>   src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/expressionOperators/GreaterThanExpr.java 704d0b8
>   src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/expressionOperators/LTOrEqualToExpr.java 9dc929e
>   src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/expressionOperators/LessThanExpr.java 0320698
>   src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/expressionOperators/Mod.java 6819185