Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig, mail # dev - Review Request 16272: [PIG-3620] TezCompiler adds duplicate predecessors of blocking operators to TezPlan


Copy link to this message
-
Review Request 16272: [PIG-3620] TezCompiler adds duplicate predecessors of blocking operators to TezPlan
Rohini Palaniswamy 2013-12-15, 20:19

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/16272/
-----------------------------------------------------------

Review request for pig, Alex Bain, Cheolsoo Park, Daniel Dai, and Mark Wagner.
Bugs: PIG-3620
    https://issues.apache.org/jira/browse/PIG-3620
Repository: pig
Description
-------

 - Removed the duplicate operators in case of split
 - Fixed multiple levels of nested splits to work
 - Added some enhancements to plan printing for easy debugging
      - Print the connectivity between the vertices in a DAG
      - Print to which Tez vertex a POLocalRearrange connects to.
      - Changed TestTezCompiler to also include the combiner optimizer to verify the combiner plan as well.
Diffs
-----

  http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/PigServer.java 1550856
  http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/POLocalRearrangeTez.java 1550856
  http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/TezCompiler.java 1550856
  http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/TezPlanContainerPrinter.java 1550856
  http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/TezPrinter.java 1550856
  http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/impl/plan/NodeIdGenerator.java 1550856
  http://svn.apache.org/repos/asf/pig/branches/tez/test/org/apache/pig/test/data/GoldenFiles/TEZC1.gld 1550856
  http://svn.apache.org/repos/asf/pig/branches/tez/test/org/apache/pig/test/data/GoldenFiles/TEZC2.gld 1550856
  http://svn.apache.org/repos/asf/pig/branches/tez/test/org/apache/pig/test/data/GoldenFiles/TEZC3.gld 1550856
  http://svn.apache.org/repos/asf/pig/branches/tez/test/org/apache/pig/test/data/GoldenFiles/TEZC4.gld 1550856
  http://svn.apache.org/repos/asf/pig/branches/tez/test/org/apache/pig/test/data/GoldenFiles/TEZC5.gld 1550856
  http://svn.apache.org/repos/asf/pig/branches/tez/test/org/apache/pig/test/data/GoldenFiles/TEZC6.gld 1550856
  http://svn.apache.org/repos/asf/pig/branches/tez/test/org/apache/pig/test/data/GoldenFiles/TEZC7.gld 1550856
  http://svn.apache.org/repos/asf/pig/branches/tez/test/org/apache/pig/test/data/GoldenFiles/TEZC8.gld 1550856
  http://svn.apache.org/repos/asf/pig/branches/tez/test/org/apache/pig/test/data/GoldenFiles/TEZC9.gld PRE-CREATION
  http://svn.apache.org/repos/asf/pig/branches/tez/test/org/apache/pig/tez/TestTezCompiler.java 1550856

Diff: https://reviews.apache.org/r/16272/diff/
Testing
-------

 - Added tests to TestTezCompiler
 - Will add the e2e tests for Split with PIG-3626. MR multi-query is also broken now. Need to fix that as well for e2e to work.
Thanks,

Rohini Palaniswamy