Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Pig >> mail # dev >> Review Request 15194: Support multiple inputs for PigProcessor


+
Mark Wagner 2013-11-02, 01:17
+
Mark Wagner 2013-11-02, 01:17
+
Cheolsoo Park 2013-11-03, 17:41
+
Cheolsoo Park 2013-11-03, 20:31
+
Daniel Dai 2013-11-05, 07:29
+
Daniel Dai 2013-11-05, 16:57
Copy link to this message
-
Re: Review Request 15194: Support multiple inputs for PigProcessor

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/15194/
-----------------------------------------------------------

(Updated Nov. 28, 2013, 12:41 a.m.)
Review request for pig, Cheolsoo Park, Daniel Dai, and Rohini Palaniswamy.
Changes
-------

Update with POPackage refactoring (see PIG-3595). There are still some items from previous reviews that I have not completed.
Bugs: PIG-3527
    https://issues.apache.org/jira/browse/PIG-3527
Repository: pig-git
Description
-------

Adds support for multiple LogicalInputs to the PigProcessor. This is done by adding a new TezLoad interface which PhysicalOperators may implement. On the backend, any operators implementing this interface will have the LogicalInput attached to them. 2 implementations are included:
* POSimpleTezLoad which consumes a single MRInput
* POShuffleTezLoad which consumes one or more ShuffledMergedInputs.
The POShuffleTezLoad does a k-way merge of the shuffle inputs to package for the operator pipeline. This required a change to the comparators used so that the sort order remained consistent. There is also a fix to POForEach where it was using the incorrect status code for signaling (although it produced the same end result in the MR pipeline).
Diffs (updated)
-----

  src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/AccumulatorOptimizer.java 3638b5c
  src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/JobControlCompiler.java fc9261c
  src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/MRCompiler.java 9341153
  src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/MRUtil.java 93de6d5
  src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/MapReduceLauncher.java 9580c26
  src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/MultiQueryOptimizer.java 64f0ee1
  src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PhyPlanSetter.java 933363d
  src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigBigDecimalRawComparator.java ddea99e
  src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigBigIntegerRawComparator.java 5ea3fc7
  src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigBooleanRawComparator.java dfd4ebf
  src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigBytesRawComparator.java 09397e5
  src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigCombiner.java 906930a
  src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigDateTimeRawComparator.java a87161f
  src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigDoubleRawComparator.java cbf457f
  src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigFloatRawComparator.java 1d86e3f
  src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigGenericMapReduce.java eea5ce3
  src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigIntRawComparator.java bb6c9df
  src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigLongRawComparator.java b3ded76
  src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigSecondaryKeyComparator.java 5ad334b
  src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigTextRawComparator.java 022f37b
  src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigTupleDefaultRawComparator.java 866c39d
  src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigTupleSortComparator.java 9724b9f
  src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/SecondaryKeyOptimizer.java 1578630
  src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/plans/POPackageAnnotator.java 47137d5
  src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/POSimpleTezLoad.java PRE-CREATION
  src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/plans/PhyPlanVisitor.java abb16ff
  src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/plans/PlanPrinter.java ff82801
  src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/plans/XMLPhysicalPlanPrinter.java 892c26f
  src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/CombinerPackager.java PRE-CREATION
  src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/JoinPackager.java PRE-CREATION
  src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/LitePackager.java PRE-CREATION
  src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/MultiQueryPackager.java PRE-CREATION
  src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/POCombinerPackage.java 9105a0e
  src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/POJoinPackage.java 82f11ac
  src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/POMultiQueryPackage.java d604174
  src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/POPackage.java 86314d9
  src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/POPackageLite.java c200715
  src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/Packager.java PRE-CREATION
  src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/util/PlanHelper.java b860521
  src/org/apache/pig/backend/hadoop/executionengine/tez/FileInputHandler.java d29e330
  src/org/apache/pig/backend/hadoop/executionengine/tez/InputHandler.java d2298ca
  src/org/apache/pig/backend/hadoop/executionengine/tez/POShuffleTezLoad.java PRE-CREATION
  src/org/apache/pig/backend/hadoop/executionengine/tez/PigProcessor.java 68610dd
  src/org/apache/pig/backend/hadoop/executionengine/tez/ShuffledInputHandler.java d7b42b8
  src/org/apache/pig/backend/hadoop/executionengine/tez/TezCompiler.java 87b198f