Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig >> mail # dev >> Review Request 17439: PIG-3728: Fix TestSkewedJoin unit test in tez mode


Copy link to this message
-
Re: Review Request 17439: PIG-3728: Fix TestSkewedJoin unit test in tez mode


> On Jan. 29, 2014, 12:39 p.m., Rohini Palaniswamy wrote:
> > src/org/apache/pig/backend/hadoop/executionengine/tez/WeightedRangePartitionerTez.java, lines 60-61
> > <https://reviews.apache.org/r/17439/diff/2/?file=452658#file452658line60>
> >
> >     In what scenario is quantiles map empty? If we don't initialize weightedParts, quantiles, etc won't we encounter NPE later?

If you sort an empty file, quantiles map will be empty. We won't encounter NPE as long as we construct a non-null but empty map in WeightRangePartitioner.
- Cheolsoo
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/17439/#review33117
-----------------------------------------------------------
On Jan. 28, 2014, 3:44 a.m., Cheolsoo Park wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/17439/
> -----------------------------------------------------------
>
> (Updated Jan. 28, 2014, 3:44 a.m.)
>
>
> Review request for pig, Alex Bain, Daniel Dai, Mark Wagner, and Rohini Palaniswamy.
>
>
> Bugs: PIG-3728
>     https://issues.apache.org/jira/browse/PIG-3728
>
>
> Repository: pig-git
>
>
> Description
> -------
>
> The patch fixes TestSkewedJoin in tez mode. Note I changed SkewedPartitionerTez and WeightedRangePartitionerTez to allow null sampleMap to make testSkewedJoinEmptyInput pass. It used to throw a runtime exception, but now it assumes empty sample input.
>
> I also fixed MiniCluster in hadoop20. They were missing the getExecType() method.
>
>
> Diffs
> -----
>
>   shims/test/hadoop20/org/apache/pig/test/MiniCluster.java 3e64a5d
>   shims/test/hadoop20/org/apache/pig/test/TezMiniCluster.java 98a580c
>   src/org/apache/pig/backend/hadoop/executionengine/tez/POPartitionRearrangeTez.java b17b38b
>   src/org/apache/pig/backend/hadoop/executionengine/tez/PigProcessor.java 2b6c899
>   src/org/apache/pig/backend/hadoop/executionengine/tez/SkewedPartitionerTez.java 5fe202b
>   src/org/apache/pig/backend/hadoop/executionengine/tez/WeightedRangePartitionerTez.java 36441fb
>   test/org/apache/pig/test/TestAccumulator.java 516d5c5
>   test/org/apache/pig/test/TestSkewedJoin.java be91d6b
>   test/tez-tests 0d32d23
>
> Diff: https://reviews.apache.org/r/17439/diff/
>
>
> Testing
> -------
>
> * TestSkewedJoin passes in both tez and mr mode.
> * ant test-tez passes except TestCustomPartitioner (known).
> * All e2e tests pass.
>
>
> Thanks,
>
> Cheolsoo Park
>
>