Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig >> mail # dev >> Review Request 14505: PIG-3501 Initial implementation of TezJobControlCompiler


Copy link to this message
-
Re: Review Request 14505: PIG-3501 Initial implementation of TezJobControlCompiler


> On Oct. 8, 2013, 1:24 a.m., Mark Wagner wrote:
> > src/org/apache/pig/backend/hadoop/executionengine/tez/TezDagBuilder.java, line 89
> > <https://reviews.apache.org/r/14505/diff/1/?file=361600#file361600line89>
> >
> >     Does it make sense to use the global conf for passing these? That could lead to bloat of the payload and the configuration is already available to Processors at initialization.

You're absolutely right. Let me create a new configuration instead. Thank you!
- Cheolsoo
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14505/#review26762
-----------------------------------------------------------
On Oct. 5, 2013, 12:28 a.m., Cheolsoo Park wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/14505/
> -----------------------------------------------------------
>
> (Updated Oct. 5, 2013, 12:28 a.m.)
>
>
> Review request for pig, Daniel Dai, Mark Wagner, and Rohini Palaniswamy.
>
>
> Bugs: PIG-3501
>     https://issues.apache.org/jira/browse/PIG-3501
>
>
> Repository: pig-git
>
>
> Description
> -------
>
> Initial implementation of TezJonControlCompiler that converts tez plan into tez dag. Tez dag is built by TezDagBuilder that is a dependency walker of TezOpPlan. At each node, it creates a tez vertex and connect it to dependent vertices with a tez edge.
>
>
> Diffs
> -----
>
>   src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/JobControlCompiler.java 8077776
>   src/org/apache/pig/backend/hadoop/executionengine/tez/DagUtils.java abd71ad
>   src/org/apache/pig/backend/hadoop/executionengine/tez/TezDagBuilder.java PRE-CREATION
>   src/org/apache/pig/backend/hadoop/executionengine/tez/TezJob.java f6a98d0
>   src/org/apache/pig/backend/hadoop/executionengine/tez/TezJobControlCompiler.java 213ef33
>   src/org/apache/pig/impl/util/Utils.java b5cffcf
>   test/org/apache/pig/tez/TestTezJobControlCompiler.java PRE-CREATION
>
> Diff: https://reviews.apache.org/r/14505/diff/
>
>
> Testing
> -------
>
> Added unit test cases to TestTezJobControlCompiler.
>
>
> Thanks,
>
> Cheolsoo Park
>
>