Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Pig >> mail # dev >> Review Request 13950: Tez backend layout


+
Cheolsoo Park 2013-09-03, 21:14
+
Cheolsoo Park 2013-09-04, 01:16
Copy link to this message
-
Re: Review Request 13950: Tez backend layout

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/13950/#review25860
-----------------------------------------------------------
This looks like a good layout to start with. There are a couple things that seem to be carried over from the 'pigeon' work, that I'm not sure are applicable.
src/org/apache/pig/backend/hadoop/executionengine/tez/MRROptimizer.java
<https://reviews.apache.org/r/13950/#comment50425>

    I think this is related to the 'pigeon' flow of compilation. Is it still applicable for the new approach?

src/org/apache/pig/backend/hadoop/executionengine/tez/TezJob.java
<https://reviews.apache.org/r/13950/#comment50426>

    Do we still need this when we have the DAG api from Tez? It seems strange to wrap Tez things in legacy MR APIs. If this is really needed, is it general enough to be included in the Tez project?

src/org/apache/pig/backend/hadoop/executionengine/tez/TezJobControlCompiler.java
<https://reviews.apache.org/r/13950/#comment50427>

    Same with above. Do we still need this? Perhaps these should be left out until the details of TezLauncher are worked out. We can add them later if it turns out we need them
- Mark Wagner
On Sept. 3, 2013, 9:14 p.m., Cheolsoo Park wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/13950/
> -----------------------------------------------------------
>
> (Updated Sept. 3, 2013, 9:14 p.m.)
>
>
> Review request for pig.
>
>
> Bugs: PIG-3448
>     https://issues.apache.org/jira/browse/PIG-3448
>
>
> Repository: pig-git
>
>
> Description
> -------
>
> Adds skeleton classes that I think we need to implement for Tez backend.
>
>
> Diffs
> -----
>
>   build.xml 7e22192
>   ivy.xml aa8f90a
>   ivy/libraries.properties 474edbd
>   src/META-INF/services/org.apache.pig.ExecType 7065767
>   src/org/apache/pig/backend/hadoop/executionengine/tez/DagUtils.java PRE-CREATION
>   src/org/apache/pig/backend/hadoop/executionengine/tez/MRROptimizer.java PRE-CREATION
>   src/org/apache/pig/backend/hadoop/executionengine/tez/MapOper.java PRE-CREATION
>   src/org/apache/pig/backend/hadoop/executionengine/tez/ReduceOper.java PRE-CREATION
>   src/org/apache/pig/backend/hadoop/executionengine/tez/TezCompiler.java PRE-CREATION
>   src/org/apache/pig/backend/hadoop/executionengine/tez/TezExecType.java PRE-CREATION
>   src/org/apache/pig/backend/hadoop/executionengine/tez/TezExecutionEngine.java PRE-CREATION
>   src/org/apache/pig/backend/hadoop/executionengine/tez/TezJob.java PRE-CREATION
>   src/org/apache/pig/backend/hadoop/executionengine/tez/TezJobControlCompiler.java PRE-CREATION
>   src/org/apache/pig/backend/hadoop/executionengine/tez/TezLauncher.java PRE-CREATION
>   src/org/apache/pig/backend/hadoop/executionengine/tez/TezOpPlanVisitor.java PRE-CREATION
>   src/org/apache/pig/backend/hadoop/executionengine/tez/TezOperPlan.java PRE-CREATION
>   src/org/apache/pig/backend/hadoop/executionengine/tez/TezOperator.java PRE-CREATION
>   src/org/apache/pig/backend/hadoop/executionengine/tez/TezPrinter.java PRE-CREATION
>
> Diff: https://reviews.apache.org/r/13950/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Cheolsoo Park
>
>

+
Mark Wagner 2013-09-03, 23:26
+
Cheolsoo Park 2013-09-03, 22:37