Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig, mail # dev - Review Request 17379: [PIG-3658] Use Tez ObjectRegistry to cache FRJoin map and WeightedRangePartitioner map


Copy link to this message
-
Re: Review Request 17379: [PIG-3658] Use Tez ObjectRegistry to cache FRJoin map and WeightedRangePartitioner map
Cheolsoo Park 2014-01-27, 18:47

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/17379/#review32854
-----------------------------------------------------------

Ship it!
Please go ahead commit this. I verified that my fix for skewed join still works with this change. I will rebase my patch to yours.

- Cheolsoo Park
On Jan. 27, 2014, 6:40 a.m., Rohini Palaniswamy wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/17379/
> -----------------------------------------------------------
>
> (Updated Jan. 27, 2014, 6:40 a.m.)
>
>
> Review request for pig, Alex Bain, Cheolsoo Park, Daniel Dai, and Mark Wagner.
>
>
> Bugs: PIG-3658
>     https://issues.apache.org/jira/browse/PIG-3658
>
>
> Repository: pig
>
>
> Description
> -------
>
> Caching broadcast inputs with vertex scope.
>
> Usage similar to hive.
> http://svn.apache.org/viewvc/hive/branches/tez/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/ObjectCache.java?view=markup
>
>
> Diffs
> -----
>
>   http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/partitioners/WeightedRangePartitioner.java 1561449
>   http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/ObjectCache.java PRE-CREATION
>   http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/POFRJoinTez.java 1561449
>   http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/POPartitionRearrangeTez.java 1561449
>   http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/PigProcessor.java 1561449
>   http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/SkewedPartitionerTez.java 1561449
>   http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/TezCompiler.java 1561449
>   http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/WeightedRangePartitionerTez.java 1561449
>
> Diff: https://reviews.apache.org/r/17379/diff/
>
>
> Testing
> -------
>
> Operators and Join e2e pass. TEZ-711 is a issue when running on scale.
>
>
> Thanks,
>
> Rohini Palaniswamy
>
>