Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Pig >> mail # dev >> Review Request 15881: PIG-3591: Refactor POPackage


Copy link to this message
-
Review Request 15881: PIG-3591: Refactor POPackage

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/15881/
-----------------------------------------------------------

Review request for pig and Cheolsoo Park.
Bugs: PIG-3591
    https://issues.apache.org/jira/browse/PIG-3591
Repository: pig-git
Description
-------

Separate "packaging" logic from "shuffle handling" logic. This moves the packaging logic to a new class "Packager", which is extended by CombinePackager, LitePackager, MultiQueryPackager, and JoinPackager.

This is not finished. Known problem are illustrate and streaming the last input are not implemented.
Diffs
-----

  src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/AccumulatorOptimizer.java 3638b5c
  src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/CombinerOptimizer.java 18a382b
  src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/JobControlCompiler.java fc9261c
  src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/MRCompiler.java 9341153
  src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/MRUtil.java 93de6d5
  src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/MapReduceLauncher.java 9580c26
  src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/MultiQueryOptimizer.java 64f0ee1
  src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PhyPlanSetter.java 933363d
  src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigCombiner.java 906930a
  src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigGenericMapReduce.java eea5ce3
  src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/SecondaryKeyOptimizer.java 1578630
  src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/plans/POPackageAnnotator.java 47137d5
  src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/plans/PhyPlanVisitor.java abb16ff
  src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/plans/PlanPrinter.java ff82801
  src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/plans/XMLPhysicalPlanPrinter.java 892c26f
  src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/CombinerPackager.java PRE-CREATION
  src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/JoinPackager.java PRE-CREATION
  src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/LitePackager.java PRE-CREATION
  src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/MultiQueryPackager.java PRE-CREATION
  src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/POCombinerPackage.java 9105a0e
  src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/POJoinPackage.java 82f11ac
  src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/POMultiQueryPackage.java d604174
  src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/POPackage.java 86314d9
  src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/POPackageLite.java c200715
  src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/Packager.java PRE-CREATION
  src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/util/PlanHelper.java b860521
  src/org/apache/pig/data/ReadOnceBag.java e2b3887
  src/org/apache/pig/newplan/logical/relational/LogToPhyTranslationVisitor.java 7112695
  src/org/apache/pig/pen/IllustratorAttacher.java db9c707
  src/org/apache/pig/pen/PhysicalPlanResetter.java f50fba7
  test/org/apache/pig/test/TestJobSubmission.java cccade4
  test/org/apache/pig/test/TestPackage.java 6b197d1
  test/org/apache/pig/test/data/GoldenFiles/Cogroup.gld 35ed75a
  test/org/apache/pig/test/data/GoldenFiles/MRC1.gld 0a34728
  test/org/apache/pig/test/data/GoldenFiles/MRC10.gld b9fad0f
  test/org/apache/pig/test/data/GoldenFiles/MRC11.gld 861608d
  test/org/apache/pig/test/data/GoldenFiles/MRC12.gld a5c5a1b
  test/org/apache/pig/test/data/GoldenFiles/MRC13.gld 0d4de2b
  test/org/apache/pig/test/data/GoldenFiles/MRC14.gld eafa509
  test/org/apache/pig/test/data/GoldenFiles/MRC15.gld cce2523
  test/org/apache/pig/test/data/GoldenFiles/MRC16.gld c319c87
  test/org/apache/pig/test/data/GoldenFiles/MRC17.gld 7e92e31
  test/org/apache/pig/test/data/GoldenFiles/MRC18.gld d13e3ab
  test/org/apache/pig/test/data/GoldenFiles/MRC19.gld 569fd21
  test/org/apache/pig/test/data/GoldenFiles/MRC2.gld 97f0ed0
  test/org/apache/pig/test/data/GoldenFiles/MRC3.gld 77a155f
  test/org/apache/pig/test/data/GoldenFiles/MRC4.gld e0bf184
  test/org/apache/pig/test/data/GoldenFiles/MRC6.gld 25dc2c3
  test/org/apache/pig/test/data/GoldenFiles/MRC7.gld 1b8e9c0
  test/org/apache/pig/test/data/GoldenFiles/MRC8.gld f3beba8
  test/org/apache/pig/test/data/GoldenFiles/MRC9.gld c162b2d

Diff: https://reviews.apache.org/r/15881/diff/
Testing

ant test-commit passes, with the exception of TestExampleGenerator which is caused by the illustrate work not being finished. I also ran some manual scripts, which worked.
Thanks,

Mark Wagner
+
Mark Wagner 2013-11-28, 00:40
+
Mark Wagner 2014-03-04, 21:40