Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig, mail # dev - Review Request 16165: PIG-3618 Replace broadcast edges with scatter/gather edges in union


Copy link to this message
-
Re: Review Request 16165: PIG-3618 Replace broadcast edges with scatter/gather edges in union
Daniel Dai 2013-12-11, 21:45

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/16165/#review30229
-----------------------------------------------------------

Ship it!
Let's put a comment subject to change after TEZ-661. waitingQueue seems unnecessary since we can remember where we are in POUnionTezLoad, and this could increase the chance of OOM. But we don't need to address it now, since it is just a temporary fix.

- Daniel Dai
On Dec. 11, 2013, 5:41 p.m., Cheolsoo Park wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/16165/
> -----------------------------------------------------------
>
> (Updated Dec. 11, 2013, 5:41 p.m.)
>
>
> Review request for pig, Alex Bain, Daniel Dai, Mark Wagner, and Rohini Palaniswamy.
>
>
> Bugs: PIG-3618
>     https://issues.apache.org/jira/browse/PIG-3618
>
>
> Repository: pig-git
>
>
> Description
> -------
>
> Replace broadcast edges with scatter/gather edges for union. I also changed the name of POPackage from POBroadcastTezLoad to POUnionTezLoad.
>
>
> Diffs
> -----
>
>   src/org/apache/pig/backend/hadoop/executionengine/tez/POBroadcastTezLoad.java d7b9d5a
>   src/org/apache/pig/backend/hadoop/executionengine/tez/POLocalRearrangeTez.java 2a96da8
>   src/org/apache/pig/backend/hadoop/executionengine/tez/POUnionTezLoad.java e69de29
>   src/org/apache/pig/backend/hadoop/executionengine/tez/PigProcessor.java 8fb18d3
>   src/org/apache/pig/backend/hadoop/executionengine/tez/TezCompiler.java ea4d167
>   src/org/apache/pig/backend/hadoop/executionengine/tez/TezDagBuilder.java 2e55ea9
>   src/org/apache/pig/backend/hadoop/executionengine/tez/TezPOPackageAnnotator.java 79c97f7
>   test/e2e/pig/tests/tez.conf b7dd862
>
> Diff: https://reviews.apache.org/r/16165/diff/
>
>
> Testing
> -------
>
> Ran tez unit tests and e2e tests.
>
> I set the parallel of union in the e2e test case to 2.
>
>
> Thanks,
>
> Cheolsoo Park
>
>