Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig, mail # dev - Review Request 17824: [PIG-3748] Support for multiquery off in Tez


Copy link to this message
-
Re: Review Request 17824: [PIG-3748] Support for multiquery off in Tez
Daniel Dai 2014-02-07, 01:09

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/17824/#review33884
-----------------------------------------------------------

Ship it!
Great you also make POValueInputTez and POValueOuputTez. Let's commit it so I can base my work on it.

- Daniel Dai
On Feb. 7, 2014, 1:03 a.m., Rohini Palaniswamy wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/17824/
> -----------------------------------------------------------
>
> (Updated Feb. 7, 2014, 1:03 a.m.)
>
>
> Review request for pig, Cheolsoo Park and Daniel Dai.
>
>
> Bugs: PIG-3748
>     https://issues.apache.org/jira/browse/PIG-3748
>
>
> Repository: pig
>
>
> Description
> -------
>
> When multiquery is off, POSplit is removed from the vertex and the tuple generated before POSplit is written once to each output and the plan after POSplit is executed in the subsequent vertex instead of being executed as a sub-plan of POSplit in the same vertex(multiquery on scenario).
>
> Optimizations TODO:
> 1) Once Tez supports shared edge, write once in POValueOutTez and let it be available for all the downstream vertices consuming the data.
> 2) POValueOutTez write key,value now with the key being empty. Weite a Input Output in Tez which only supports values and avoid writing empty keys.
>
>
> Diffs
> -----
>
>   http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/Main.java 1565502
>   http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/PigServer.java 1565502
>   http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/MapReduceLauncher.java 1565502
>   http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/plans/PhyPlanVisitor.java 1565502
>   http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/util/PlanHelper.java 1565502
>   http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/POValueInputTez.java PRE-CREATION
>   http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/POValueOutputTez.java PRE-CREATION
>   http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/PigProcessor.java 1565502
>   http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/TezCompiler.java 1565502
>   http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/TezDagBuilder.java 1565502
>   http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/TezOperPlan.java 1565502
>   http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/TezOperator.java 1565502
>   http://svn.apache.org/repos/asf/pig/branches/tez/test/org/apache/pig/test/TestBatchAliases.java 1565502
>   http://svn.apache.org/repos/asf/pig/branches/tez/test/org/apache/pig/test/TestGrunt.java 1565502
>   http://svn.apache.org/repos/asf/pig/branches/tez/test/org/apache/pig/test/TestLoad.java 1565502
>   http://svn.apache.org/repos/asf/pig/branches/tez/test/org/apache/pig/test/TestMultiQuery.java 1565502
>   http://svn.apache.org/repos/asf/pig/branches/tez/test/org/apache/pig/test/TestMultiQueryBasic.java 1565502
>   http://svn.apache.org/repos/asf/pig/branches/tez/test/org/apache/pig/test/TestMultiQueryCompiler.java 1565502
>   http://svn.apache.org/repos/asf/pig/branches/tez/test/org/apache/pig/test/TestMultiQueryLocal.java 1565502
>   http://svn.apache.org/repos/asf/pig/branches/tez/test/org/apache/pig/test/TestPigRunner.java 1565502
>   http://svn.apache.org/repos/asf/pig/branches/tez/test/org/apache/pig/test/TestPigServer.java 1565502