Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Pig, mail # dev - Review Request: Add BigInteger and BigDecimal to Pig


+
Jonathan Coveney 2013-01-17, 21:02
+
Mathias Herberts 2013-01-17, 23:16
+
Jonathan Coveney 2013-01-18, 22:11
+
Mathias Herberts 2013-01-19, 19:29
+
Jonathan Coveney 2013-01-22, 19:31
Copy link to this message
-
Re: Review Request: Add BigInteger and BigDecimal to Pig
Alan Gates 2013-01-21, 18:27

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/9012/#review15544
-----------------------------------------------------------
Looks good.  The only comment that I think needs to be addressed before checking it in is the one on TypeAwareTuple.java.  After that I'm +1 on checking this in.  The rest are questions or JIRAs can be filed for later.

I also think we should file a JIRA to add e2e tests for these two types and make it a blocker for 0.12.  I know with the datetime stuff we found a bunch of new issues when we added those tests.

As a general question, I'm trying to figure out the utility of BigInteger.  I totally get BigDecimal, since people want fixed precision calculations.  BigInteger would be useful if you want integer semantics for numbers larger than 2^63, but I can't think of a use case for that off the top of my head.  I'm not saying we shouldn't do it, I just wonder if it will be used.
src/org/apache/pig/backend/hadoop/hbase/HBaseBinaryConverter.java
<https://reviews.apache.org/r/9012/#comment33547>

    Is this because HBase doesn't have a standard representation for this?

src/org/apache/pig/builtin/BinStorage.java
<https://reviews.apache.org/r/9012/#comment33548>

    Why not implement these?  You have read and write functions in the BytesWritable implementations.

src/org/apache/pig/builtin/TextLoader.java
<https://reviews.apache.org/r/9012/#comment33549>

    Again, why not implement these?  There are a string to bigint and bigdecimal functions.

src/org/apache/pig/data/BinInterSedes.java
<https://reviews.apache.org/r/9012/#comment33550>

    I wonder if there's a way to avoid conversions to and from and strings here.  That can hardly be efficient.  I don't think it's something we need to change now but down the road maybe something to think about.

src/org/apache/pig/data/TypeAwareTuple.java
<https://reviews.apache.org/r/9012/#comment33551>

    Seems like the second argument here should be a BigInteger, not a boolean.  Same comment for the next line.
- Alan Gates
On Jan. 18, 2013, 10:11 p.m., Jonathan Coveney wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/9012/
> -----------------------------------------------------------
>
> (Updated Jan. 18, 2013, 10:11 p.m.)
>
>
> Review request for pig, Alan Gates and Mathias Herberts.
>
>
> Description
> -------
>
> This patch adds big integer and big decimal support to Pig. It could use more tests, something I'd appreciate feedback on (but I wanted to make sure the core implementation is good)
>
>
> This addresses bug PIG-2764.
>     https://issues.apache.org/jira/browse/PIG-2764
>
>
> Diffs
> -----
>
>   .gitignore cc62d7d
>   src/org/apache/pig/LoadCaster.java 574769b
>   src/org/apache/pig/PigWarning.java 5de075f
>   src/org/apache/pig/StoreCaster.java 5fe48de
>   src/org/apache/pig/backend/hadoop/BigDecimalWritable.java PRE-CREATION
>   src/org/apache/pig/backend/hadoop/BigIntegerWritable.java PRE-CREATION
>   src/org/apache/pig/backend/hadoop/HDataType.java 84a56b8
>   src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/JobControlCompiler.java 96fba6b
>   src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigBigDecimalRawComparator.java PRE-CREATION
>   src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigBigIntegerRawComparator.java PRE-CREATION
>   src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/partitioners/WeightedRangePartitioner.java 9749339
>   src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/PhysicalOperator.java f40eb43
>   src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/expressionOperators/Add.java c84b767
>   src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/expressionOperators/ConstantExpression.java db3840f
+
Jonathan Coveney 2013-01-22, 19:31
+
Cheolsoo Park 2013-01-20, 23:34
+
Jonathan Coveney 2013-01-22, 22:06
+
Mathias Herberts 2013-01-22, 22:08
+
Mathias Herberts 2013-01-22, 22:08
+
Russell Jurney 2013-01-25, 20:19
+
Jonathan Coveney 2013-01-18, 21:47