Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig, mail # dev - Review Request 14801: PIG-3531 Add TezMiniCluster for unit tests


Copy link to this message
-
Re: Review Request 14801: PIG-3531 Add TezMiniCluster for unit tests
Rohini Palaniswamy 2013-10-22, 17:54

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14801/#review27313
-----------------------------------------------------------
Would be good to avoid mentioning TezMiniCluster in test code. What I am thinking is something like this - invoking tests with -Dhadoopversion=23 -Dexecmode=tez, should run the whole test suite using tez mode. We can and will have individual tests for MR and Tez still which will not consider -execmode and TezMiniCluster can be directly used in them. If MiniGenericCluster.buildCluster returns TezMiniCluster if -Dexecmode=tez, then it will be simpler and we don't have to change the current declaration of "private static MiniCluster cluster = MiniCluster.buildCluster();" in all the test classes for now. When we do a merge with trunk, we can change that to MiniGenericCluster.buildCluster(); to be cleaner.

- Rohini Palaniswamy
On Oct. 22, 2013, 1:02 a.m., Cheolsoo Park wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/14801/
> -----------------------------------------------------------
>
> (Updated Oct. 22, 2013, 1:02 a.m.)
>
>
> Review request for pig, Daniel Dai, Mark Wagner, and Rohini Palaniswamy.
>
>
> Bugs: PIG-3531
>     https://issues.apache.org/jira/browse/PIG-3531
>
>
> Repository: pig-git
>
>
> Description
> -------
>
> Add TezMiniCluster that allows us to write unit tests. The TezMiniCluster class extends MiniGenericCluster and set up Tez-specific properties and jars.
>
>
> Diffs
> -----
>
>   shims/test/hadoop20/org/apache/pig/test/MiniCluster.java d6384e9
>   shims/test/hadoop23/org/apache/pig/test/MiniCluster.java 1a4a430
>   shims/test/hadoop23/org/apache/pig/test/TezMiniCluster.java e69de29
>   test/org/apache/pig/test/MiniGenericCluster.java ac3f5bc
>   test/org/apache/pig/test/Util.java 24103da
>   test/org/apache/pig/tez/TestTezLauncher.java d4c3c6a
>
> Diff: https://reviews.apache.org/r/14801/diff/
>
>
> Testing
> -------
>
> Updated TestTezLauncher using TezMiniCluster. Now it asserts the output is the same as what's expected.
>
>
> Thanks,
>
> Cheolsoo Park
>
>