Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig, mail # dev - Review Request 17439: PIG-3728: Fix TestSkewedJoin unit test in tez mode


Copy link to this message
-
Re: Review Request 17439: PIG-3728: Fix TestSkewedJoin unit test in tez mode
Rohini Palaniswamy 2014-01-29, 12:39

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/17439/#review33117
-----------------------------------------------------------

src/org/apache/pig/backend/hadoop/executionengine/tez/WeightedRangePartitionerTez.java
<https://reviews.apache.org/r/17439/#comment62402>

    In what scenario is quantiles map empty? If we don't initialize weightedParts, quantiles, etc won't we encounter NPE later?
- Rohini Palaniswamy
On Jan. 28, 2014, 3:44 a.m., Cheolsoo Park wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/17439/
> -----------------------------------------------------------
>
> (Updated Jan. 28, 2014, 3:44 a.m.)
>
>
> Review request for pig, Alex Bain, Daniel Dai, Mark Wagner, and Rohini Palaniswamy.
>
>
> Bugs: PIG-3728
>     https://issues.apache.org/jira/browse/PIG-3728
>
>
> Repository: pig-git
>
>
> Description
> -------
>
> The patch fixes TestSkewedJoin in tez mode. Note I changed SkewedPartitionerTez and WeightedRangePartitionerTez to allow null sampleMap to make testSkewedJoinEmptyInput pass. It used to throw a runtime exception, but now it assumes empty sample input.
>
> I also fixed MiniCluster in hadoop20. They were missing the getExecType() method.
>
>
> Diffs
> -----
>
>   shims/test/hadoop20/org/apache/pig/test/MiniCluster.java 3e64a5d
>   shims/test/hadoop20/org/apache/pig/test/TezMiniCluster.java 98a580c
>   src/org/apache/pig/backend/hadoop/executionengine/tez/POPartitionRearrangeTez.java b17b38b
>   src/org/apache/pig/backend/hadoop/executionengine/tez/PigProcessor.java 2b6c899
>   src/org/apache/pig/backend/hadoop/executionengine/tez/SkewedPartitionerTez.java 5fe202b
>   src/org/apache/pig/backend/hadoop/executionengine/tez/WeightedRangePartitionerTez.java 36441fb
>   test/org/apache/pig/test/TestAccumulator.java 516d5c5
>   test/org/apache/pig/test/TestSkewedJoin.java be91d6b
>   test/tez-tests 0d32d23
>
> Diff: https://reviews.apache.org/r/17439/diff/
>
>
> Testing
> -------
>
> * TestSkewedJoin passes in both tez and mr mode.
> * ant test-tez passes except TestCustomPartitioner (known).
> * All e2e tests pass.
>
>
> Thanks,
>
> Cheolsoo Park
>
>