Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig, mail # dev - Re: Review Request 15881: PIG-3591: Refactor POPackage


Copy link to this message
-
Re: Review Request 15881: PIG-3591: Refactor POPackage
Mark Wagner 2014-04-05, 02:25

This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/15881/

(Updated April 5, 2014, 2:25 a.m.)
Review request for pig and Cheolsoo Park.
Bugs: PIG-3591
    https://issues.apache.org/jira/browse/PIG-3591
Repository: pig-git
Description

Separate "packaging" logic from "shuffle handling" logic. This moves the packaging logic to a new class "Packager", which is extended by CombinePackager, LitePackager, MultiQueryPackager, and JoinPackager.

This is not finished. Known problem are illustrate and streaming the last input are not implemented.
Diffs (updated)

  src/org/apache/pig/backend/hadoop/executionengine/fetch/FetchOptimizer.java d801f6f
  src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/AccumulatorOptimizer.java 3638b5c
  src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/CombinerOptimizer.java 18a382b
  src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/JobControlCompiler.java b7ce7e2
  src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/MRCompiler.java 5dddab7
  src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/MRUtil.java 93de6d5
  src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/MapReduceLauncher.java d3ebeb3
  src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/MultiQueryOptimizer.java 64f0ee1
  src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PhyPlanSetter.java 933363d
  src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigCombiner.java 773a22c
  src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigGenericMapReduce.java eea5ce3
  src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/SecondaryKeyOptimizer.java 54740a0
  src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/plans/POPackageAnnotator.java 47137d5
  src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/plans/PhyPlanVisitor.java abb16ff
  src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/plans/PlanPrinter.java ff82801
  src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/plans/XMLPhysicalPlanPrinter.java 892c26f
  src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/CombinerPackager.java PRE-CREATION
  src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/JoinPackager.java PRE-CREATION
  src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/LitePackager.java PRE-CREATION
  src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/MultiQueryPackager.java PRE-CREATION
  src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/POCombinerPackage.java 9105a0e
  src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/POJoinPackage.java 82f11ac
  src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/POMultiQueryPackage.java d604174
  src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/POPackage.java 86314d9
  src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/POPackageLite.java c200715
  src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/POPartialAgg.java a92e643
  src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/Packager.java PRE-CREATION
  src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/util/PlanHelper.java b860521
  src/org/apache/pig/data/ReadOnceBag.java e2b3887
  src/org/apache/pig/data/SelfSpillBag.java 2fcf36a
  src/org/apache/pig/newplan/logical/relational/LogToPhyTranslationVisitor.java 7112695
  src/org/apache/pig/pen/IllustratorAttacher.java db9c707
  src/org/apache/pig/pen/PhysicalPlanResetter.java f50fba7
  test/org/apache/pig/test/TestExampleGenerator.java 42f757f
  test/org/apache/pig/test/TestJobSubmission.java 9741dc9
  test/org/apache/pig/test/TestPackage.java 6b197d1
  test/org/apache/pig/test/data/GoldenFiles/Cogroup.gld 35ed75a
  test/org/apache/pig/test/data/GoldenFiles/MRC1.gld 0a34728
  test/org/apache/pig/test/data/GoldenFiles/MRC10.gld b9fad0f
  test/org/apache/pig/test/data/GoldenFiles/MRC11.gld 861608d
  test/org/apache/pig/test/data/GoldenFiles/MRC12.gld 15f1158
  test/org/apache/pig/test/data/GoldenFiles/MRC13.gld 0d4de2b
  test/org/apache/pig/test/data/GoldenFiles/MRC14.gld eafa509
  test/org/apache/pig/test/data/GoldenFiles/MRC15.gld e8bd281
  test/org/apache/pig/test/data/GoldenFiles/MRC16.gld 6151d1e
  test/org/apache/pig/test/data/GoldenFiles/MRC17.gld cec5d57
  test/org/apache/pig/test/data/GoldenFiles/MRC18.gld 01cdc32
  test/org/apache/pig/test/data/GoldenFiles/MRC19.gld 8688178
  test/org/apache/pig/test/data/GoldenFiles/MRC2.gld 97f0ed0
  test/org/apache/pig/test/data/GoldenFiles/MRC3.gld 77a155f
  test/org/apache/pig/test/data/GoldenFiles/MRC4.gld 7c5078e
  test/org/apache/pig/test/data/GoldenFiles/MRC6.gld 45cba2d
  test/org/apache/pig/test/data/GoldenFiles/MRC7.gld 09fcaec
  test/org/apache/pig/test/data/GoldenFiles/MRC8.gld 2d301a9
  test/org/apache/pig/test/data/GoldenFiles/MRC9.gld c5d7047

Diff: https://reviews.apache.org/r/15881/diff/
Testing

ant test-commit passes, with the exception of TestExampleGenerator which is caused by the illustrate work not being finished. I also ran some manual scripts, which worked.
Thanks,

Mark Wagner