Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig >> mail # dev >> Review Request: Add BigInteger and BigDecimal to Pig


Copy link to this message
-
Re: Review Request: Add BigInteger and BigDecimal to Pig


> On Jan. 21, 2013, 6:27 p.m., Alan Gates wrote:
> > src/org/apache/pig/backend/hadoop/hbase/HBaseBinaryConverter.java, line 192
> > <https://reviews.apache.org/r/9012/diff/2/?file=250198#file250198line192>
> >
> >     Is this because HBase doesn't have a standard representation for this?

Yes, that's how I understood it
> On Jan. 21, 2013, 6:27 p.m., Alan Gates wrote:
> > src/org/apache/pig/builtin/BinStorage.java, line 143
> > <https://reviews.apache.org/r/9012/diff/2/?file=250203#file250203line143>
> >
> >     Why not implement these?  You have read and write functions in the BytesWritable implementations.

It's specifically a load caster that has nothing implemented
> On Jan. 21, 2013, 6:27 p.m., Alan Gates wrote:
> > src/org/apache/pig/builtin/TextLoader.java, line 249
> > <https://reviews.apache.org/r/9012/diff/2/?file=250204#file250204line249>
> >
> >     Again, why not implement these?  There are a string to bigint and bigdecimal functions.

In this case it is because of how the TextLoader works. The whole point of the textloader is that you don't try to cast anything, and you just load each line as text. Note that the only implemented one is bytesToCharArray
> On Jan. 21, 2013, 6:27 p.m., Alan Gates wrote:
> > src/org/apache/pig/data/BinInterSedes.java, line 908
> > <https://reviews.apache.org/r/9012/diff/2/?file=250206#file250206line908>
> >
> >     I wonder if there's a way to avoid conversions to and from and strings here.  That can hardly be efficient.  I don't think it's something we need to change now but down the road maybe something to think about.

I wholeheartedly agree. I think we should make a JIRA for this. I didn't sweat it since raging efficiency isn't what this is about anyway, but we definitely could do something much smarter. It would not be hard.
> On Jan. 21, 2013, 6:27 p.m., Alan Gates wrote:
> > src/org/apache/pig/data/TypeAwareTuple.java, line 39
> > <https://reviews.apache.org/r/9012/diff/2/?file=250211#file250211line39>
> >
> >     Seems like the second argument here should be a BigInteger, not a boolean.  Same comment for the next line.

Oh wow lol. Copy-paste strikes. Good catch.
- Jonathan
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/9012/#review15544
-----------------------------------------------------------
On Jan. 18, 2013, 10:11 p.m., Jonathan Coveney wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/9012/
> -----------------------------------------------------------
>
> (Updated Jan. 18, 2013, 10:11 p.m.)
>
>
> Review request for pig, Alan Gates and Mathias Herberts.
>
>
> Description
> -------
>
> This patch adds big integer and big decimal support to Pig. It could use more tests, something I'd appreciate feedback on (but I wanted to make sure the core implementation is good)
>
>
> This addresses bug PIG-2764.
>     https://issues.apache.org/jira/browse/PIG-2764
>
>
> Diffs
> -----
>
>   .gitignore cc62d7d
>   src/org/apache/pig/LoadCaster.java 574769b
>   src/org/apache/pig/PigWarning.java 5de075f
>   src/org/apache/pig/StoreCaster.java 5fe48de
>   src/org/apache/pig/backend/hadoop/BigDecimalWritable.java PRE-CREATION
>   src/org/apache/pig/backend/hadoop/BigIntegerWritable.java PRE-CREATION
>   src/org/apache/pig/backend/hadoop/HDataType.java 84a56b8
>   src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/JobControlCompiler.java 96fba6b
>   src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigBigDecimalRawComparator.java PRE-CREATION
>   src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigBigIntegerRawComparator.java PRE-CREATION
>   src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/partitioners/WeightedRangePartitioner.java 9749339
>   src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/PhysicalOperator.java f40eb43