Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig >> mail # dev >> Review Request 17824: [PIG-3748] Support for multiquery off in Tez


Copy link to this message
-
Review Request 17824: [PIG-3748] Support for multiquery off in Tez

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/17824/
-----------------------------------------------------------

Review request for pig, Cheolsoo Park and Daniel Dai.
Bugs: PIG-3748
    https://issues.apache.org/jira/browse/PIG-3748
Repository: pig
Description
-------

When multiquery is off, POSplit is removed from the vertex and the tuple generated before POSplit is written once to each output and the plan after POSplit is executed in the subsequent vertex instead of being executed as a sub-plan of POSplit in the same vertex(multiquery on scenario).

Optimizations TODO:
1) Once Tez supports shared edge, write once in POValueOutTez and let it be available for all the downstream vertices consuming the data.
2) POValueOutTez write key,value now with the key being empty. Weite a Input Output in Tez which only supports values and avoid writing empty keys.
Diffs
-----

  http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/Main.java 1565502
  http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/PigServer.java 1565502
  http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/MapReduceLauncher.java 1565502
  http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/plans/PhyPlanVisitor.java 1565502
  http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/util/PlanHelper.java 1565502
  http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/POValueInputTez.java PRE-CREATION
  http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/POValueOutputTez.java PRE-CREATION
  http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/PigProcessor.java 1565502
  http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/TezCompiler.java 1565502
  http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/TezDagBuilder.java 1565502
  http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/TezOperPlan.java 1565502
  http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/TezOperator.java 1565502
  http://svn.apache.org/repos/asf/pig/branches/tez/test/org/apache/pig/test/TestBatchAliases.java 1565502
  http://svn.apache.org/repos/asf/pig/branches/tez/test/org/apache/pig/test/TestGrunt.java 1565502
  http://svn.apache.org/repos/asf/pig/branches/tez/test/org/apache/pig/test/TestLoad.java 1565502
  http://svn.apache.org/repos/asf/pig/branches/tez/test/org/apache/pig/test/TestMultiQuery.java 1565502
  http://svn.apache.org/repos/asf/pig/branches/tez/test/org/apache/pig/test/TestMultiQueryBasic.java 1565502
  http://svn.apache.org/repos/asf/pig/branches/tez/test/org/apache/pig/test/TestMultiQueryCompiler.java 1565502
  http://svn.apache.org/repos/asf/pig/branches/tez/test/org/apache/pig/test/TestMultiQueryLocal.java 1565502
  http://svn.apache.org/repos/asf/pig/branches/tez/test/org/apache/pig/test/TestPigRunner.java 1565502
  http://svn.apache.org/repos/asf/pig/branches/tez/test/org/apache/pig/test/TestPigServer.java 1565502
  http://svn.apache.org/repos/asf/pig/branches/tez/test/org/apache/pig/test/TestStore.java 1565502
  http://svn.apache.org/repos/asf/pig/branches/tez/test/org/apache/pig/test/data/GoldenFiles/TEZC7.gld 1565502

Diff: https://reviews.apache.org/r/17824/diff/
Testing
-------

Manually tested. Will fix older unit tests or add new ones later.

Also Updated TezC7.gld file which was failing because of some earlier jira changes.
Thanks,

Rohini Palaniswamy