Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig, mail # dev - Review Request 17529: [PIG-3732] Use ONE_TO_ONE edge and IdentityInOut in orderby intermediate vertex


Copy link to this message
-
Re: Review Request 17529: [PIG-3732] Use ONE_TO_ONE edge and IdentityInOut in orderby intermediate vertex
Daniel Dai 2014-01-30, 19:53

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/17529/#review33243
-----------------------------------------------------------

http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/TezCompiler.java
<https://reviews.apache.org/r/17529/#comment62625>

    What's the reason for this change? lr should be already constructed, right?
- Daniel Dai
On Jan. 30, 2014, 8:28 a.m., Rohini Palaniswamy wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/17529/
> -----------------------------------------------------------
>
> (Updated Jan. 30, 2014, 8:28 a.m.)
>
>
> Review request for pig, Cheolsoo Park and Daniel Dai.
>
>
> Bugs: PIG-3732
>     https://issues.apache.org/jira/browse/PIG-3732
>
>
> Repository: pig
>
>
> Description
> -------
>
> Orderby has 4 vertices and changes done are as below.
>
> Load Vertex -> Partitioner Vertex
>      - Was RoundRobinPartitioner with sorted shuffle and parallelism of Partitioner Vertex was same as reducer vertex (i.e PARALLEL clause). Now ONE_TO_ONE unsorted edge between Load Vertex and Partitioner Vertex with Partitioner Vertex having same parallelism as Load Vertex. Will get the performance numbers for both cases by Friday.
> Load Vertex -> Sampler Vertex  
> Sampler Vertex -> Partitioner Vertex (Broadcast edge)
>      - The POPackage->POForeach->POLocalRearrange in Partitioner Vertex has been replaced by POIdentityInOutTez
> Partitioner Vertex -> Reducer Vertex
>
> Need to attempt this for Skewed Join as well.
>
>
> This patch also sets credential on DAG which is required after TEZ-395
>
>
> Diffs
> -----
>
>   http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/partitioners/WeightedRangePartitioner.java 1562426
>   http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/PhysicalOperator.java 1562426
>   http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/POIdentityInOutTez.java PRE-CREATION
>   http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/POLocalRearrangeTez.java 1562426
>   http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/POPartitionRearrangeTez.java 1562426
>   http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/POShuffleTezLoad.java 1562426
>   http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/PigProcessor.java 1562426
>   http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/TezCompiler.java 1562426
>   http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/TezDAG.java 1562426
>   http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/WeightedRangePartitionerTez.java 1562426
>   http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/impl/io/NullablePartitionWritable.java 1562426
>   http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/impl/io/PigNullableWritable.java 1562426
>   http://svn.apache.org/repos/asf/pig/branches/tez/test/org/apache/pig/test/data/GoldenFiles/TEZC16.gld 1562426
>   http://svn.apache.org/repos/asf/pig/branches/tez/test/org/apache/pig/test/data/GoldenFiles/TEZC7.gld 1562426
>
> Diff: https://reviews.apache.org/r/17529/diff/
>
>
> Testing
> -------
>
> test-tez and tez.conf e2e tests pass
>
>
> Thanks,
>
> Rohini Palaniswamy
>
>