Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig, mail # dev - Review Request 14505: PIG-3501 Initial implementation of TezJobControlCompiler


Copy link to this message
-
Re: Review Request 14505: PIG-3501 Initial implementation of TezJobControlCompiler
Mark Wagner 2013-10-08, 01:24

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14505/#review26762
-----------------------------------------------------------

src/org/apache/pig/backend/hadoop/executionengine/tez/TezDagBuilder.java
<https://reviews.apache.org/r/14505/#comment52091>

    Does it make sense to use the global conf for passing these? That could lead to bloat of the payload and the configuration is already available to Processors at initialization.

src/org/apache/pig/backend/hadoop/executionengine/tez/TezJob.java
<https://reviews.apache.org/r/14505/#comment52090>

    getAppId?

src/org/apache/pig/backend/hadoop/executionengine/tez/TezJobControlCompiler.java
<https://reviews.apache.org/r/14505/#comment52092>

    Is this our responsibility? I understood verify to be mostly be for Tez internal. Not a big deal either way though.
- Mark Wagner
On Oct. 5, 2013, 12:28 a.m., Cheolsoo Park wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/14505/
> -----------------------------------------------------------
>
> (Updated Oct. 5, 2013, 12:28 a.m.)
>
>
> Review request for pig, Daniel Dai, Mark Wagner, and Rohini Palaniswamy.
>
>
> Bugs: PIG-3501
>     https://issues.apache.org/jira/browse/PIG-3501
>
>
> Repository: pig-git
>
>
> Description
> -------
>
> Initial implementation of TezJonControlCompiler that converts tez plan into tez dag. Tez dag is built by TezDagBuilder that is a dependency walker of TezOpPlan. At each node, it creates a tez vertex and connect it to dependent vertices with a tez edge.
>
>
> Diffs
> -----
>
>   src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/JobControlCompiler.java 8077776
>   src/org/apache/pig/backend/hadoop/executionengine/tez/DagUtils.java abd71ad
>   src/org/apache/pig/backend/hadoop/executionengine/tez/TezDagBuilder.java PRE-CREATION
>   src/org/apache/pig/backend/hadoop/executionengine/tez/TezJob.java f6a98d0
>   src/org/apache/pig/backend/hadoop/executionengine/tez/TezJobControlCompiler.java 213ef33
>   src/org/apache/pig/impl/util/Utils.java b5cffcf
>   test/org/apache/pig/tez/TestTezJobControlCompiler.java PRE-CREATION
>
> Diff: https://reviews.apache.org/r/14505/diff/
>
>
> Testing
> -------
>
> Added unit test cases to TestTezJobControlCompiler.
>
>
> Thanks,
>
> Cheolsoo Park
>
>