Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig, mail # dev - Review Request 17529: [PIG-3732] Use ONE_TO_ONE edge and IdentityInOut in orderby intermediate vertex


Copy link to this message
-
Re: Review Request 17529: [PIG-3732] Use ONE_TO_ONE edge and IdentityInOut in orderby intermediate vertex
Rohini Palaniswamy 2014-01-31, 00:22

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/17529/
-----------------------------------------------------------

(Updated Jan. 31, 2014, 12:21 a.m.)
Review request for pig, Cheolsoo Park and Daniel Dai.
Changes
-------

Final patch that was committed
Bugs: PIG-3732
    https://issues.apache.org/jira/browse/PIG-3732
Repository: pig
Description
-------

Orderby has 4 vertices and changes done are as below.

Load Vertex -> Partitioner Vertex
     - Was RoundRobinPartitioner with sorted shuffle and parallelism of Partitioner Vertex was same as reducer vertex (i.e PARALLEL clause). Now ONE_TO_ONE unsorted edge between Load Vertex and Partitioner Vertex with Partitioner Vertex having same parallelism as Load Vertex. Will get the performance numbers for both cases by Friday.
Load Vertex -> Sampler Vertex  
Sampler Vertex -> Partitioner Vertex (Broadcast edge)
     - The POPackage->POForeach->POLocalRearrange in Partitioner Vertex has been replaced by POIdentityInOutTez
Partitioner Vertex -> Reducer Vertex

Need to attempt this for Skewed Join as well.
This patch also sets credential on DAG which is required after TEZ-395
Diffs (updated)
-----

  http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/partitioners/WeightedRangePartitioner.java 1562426
  http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/PhysicalOperator.java 1562426
  http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/POIdentityInOutTez.java PRE-CREATION
  http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/POLocalRearrangeTez.java 1562426
  http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/POPartitionRearrangeTez.java 1562426
  http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/POShuffleTezLoad.java 1562426
  http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/PigProcessor.java 1562426
  http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/TezCompiler.java 1562426
  http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/TezDAG.java 1562426
  http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/WeightedRangePartitionerTez.java 1562426
  http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/impl/io/NullablePartitionWritable.java 1562426
  http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/impl/io/PigNullableWritable.java 1562426
  http://svn.apache.org/repos/asf/pig/branches/tez/test/org/apache/pig/test/data/GoldenFiles/TEZC16.gld 1562426
  http://svn.apache.org/repos/asf/pig/branches/tez/test/org/apache/pig/test/data/GoldenFiles/TEZC7.gld 1562426

Diff: https://reviews.apache.org/r/17529/diff/
Testing
-------

test-tez and tez.conf e2e tests pass
Thanks,

Rohini Palaniswamy