Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig >> mail # dev >> Review Request: Add BigInteger and BigDecimal to Pig

Copy link to this message
Re: Review Request: Add BigInteger and BigDecimal to Pig

This is an automatically generated e-mail. To reply, visit:


    I'm not sure what the contract of HBaseBinaryConverter is, which is why I didn't implement it. Theoretically, any Pig type can be serialized to Bytes since we do this in BinInterSedes, but here they were very particular to only use HBase's serialization, leading me to believe we should only convert bytes in the case where they can be transformed into the HBase analogue. We should find someone who knows more about this code, or dig around to see how it is used.


    None of the other calls in this class specified the charset, so I did not either. Is there a reason why in this case specifically we need to, given we did bi => string => bytes?


    Odd. I think this is because of the preexisting presence of tabs in the file, which I will remove.


    This is janky, but to understand what is going on you need to look at DataReaderWriter to see how BigInteger and BigDecimal are being serialized in this specific case (which is not BinInterSedes, and in fact, I don't think this code is called anywhere...but still, it should be correct. We should probably unify all the serialization under BinInterSedes if possible, but that's a JIRA for another day). So basically, in this case we serialize as a String, and then decode. I'm going to update it to serialize the bytes directly...no reason not to.


    Thus the TODO :S I completely punted on figuring out how big they are. I will try and tackle that now...thanks for the link, it will definitely help a lot


    Same comment here as earlier. It may need to be, but I'm unsure, and if it does, why doesn't it need to be in the other cases?
- Jonathan Coveney
On Jan. 17, 2013, 9:02 p.m., Jonathan Coveney wrote:
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/9012/
> -----------------------------------------------------------
> (Updated Jan. 17, 2013, 9:02 p.m.)
> Review request for pig, Alan Gates and Mathias Herberts.
> Description
> -------
> This patch adds big integer and big decimal support to Pig. It could use more tests, something I'd appreciate feedback on (but I wanted to make sure the core implementation is good)
> This addresses bug PIG-2764.
>     https://issues.apache.org/jira/browse/PIG-2764
> Diffs
> -----
>   .gitignore cc62d7d
>   src/org/apache/pig/LoadCaster.java 574769b
>   src/org/apache/pig/PigWarning.java 5de075f
>   src/org/apache/pig/StoreCaster.java 5fe48de
>   src/org/apache/pig/backend/hadoop/BigDecimalWritable.java PRE-CREATION
>   src/org/apache/pig/backend/hadoop/BigIntegerWritable.java PRE-CREATION
>   src/org/apache/pig/backend/hadoop/HDataType.java 84a56b8
>   src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/JobControlCompiler.java 96fba6b
>   src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigBigDecimalRawComparator.java PRE-CREATION
>   src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigBigIntegerRawComparator.java PRE-CREATION
>   src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/partitioners/WeightedRangePartitioner.java 9749339