Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig >> mail # dev >> Review Request 17191: [PIG-3626] Make combiners, custom partitioners and secondary key sort work for multiple outputs


Copy link to this message
-
Re: Review Request 17191: [PIG-3626] Make combiners, custom partitioners and secondary key sort work for multiple outputs


> On Jan. 22, 2014, 6:43 p.m., Cheolsoo Park wrote:
> > I am reviewing the patch now.
> >
> > Thank you so much for fixing skewed join e2e tests. I was struggling with them for few days!

Talked to Rohini. I misunderstood her comment about skewed join e2e tests. She was referring to tez.conf tests while I was referring to nightly.conf tests.

To summarize,
1) tez.conf Join_7,8 are passing in current tez branch.
2) tez.conf Join_7,8 are broken with this patch but Rohini will fix them.
3) nightly.conf SkewedJoin are broken in current tez branch. I am still debugging them.
- Cheolsoo
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/17191/#review32513
-----------------------------------------------------------
On Jan. 22, 2014, 6:18 p.m., Rohini Palaniswamy wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/17191/
> -----------------------------------------------------------
>
> (Updated Jan. 22, 2014, 6:18 p.m.)
>
>
> Review request for pig, Alex Bain, Cheolsoo Park, Daniel Dai, and Mark Wagner.
>
>
> Repository: pig
>
>
> Description
> -------
>
> Added support for secondary key sort. Also fixed combiners and custom partitioners to work for multiple outputs. Fixed bugs in couple of places as I encountered them.
>
>
> Diffs
> -----
>
>   http://svn.apache.org/repos/asf/pig/branches/tez/shims/test/hadoop23/org/apache/pig/test/MiniCluster.java 1559126
>   http://svn.apache.org/repos/asf/pig/branches/tez/shims/test/hadoop23/org/apache/pig/test/TezMiniCluster.java 1559126
>   http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/PigConfiguration.java 1559126
>   http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/ColumnInfo.java 1559126
>   http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/MapReduceLauncher.java 1559126
>   http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/SecondaryKeyOptimizerMR.java PRE-CREATION
>   http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/optimizer/SecondaryKeyOptimizer.java PRE-CREATION
>   http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/PhysicalOperator.java 1559126
>   http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/POLocalRearrange.java 1559126
>   http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/POReservoirSample.java 1559126
>   http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/util/PlanHelper.java 1559126
>   http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/CombinerOptimizer.java 1559126
>   http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/POShuffleTezLoad.java 1559126
>   http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/SecondaryKeyOptimizerTez.java PRE-CREATION
>   http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/SkewedPartitionerTez.java 1559126
>   http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/TezCompiler.java 1559126
>   http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/TezCompilerUtil.java 1559126
>   http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/TezDagBuilder.java 1559126