Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Sqoop, mail # dev - Review Request: SQOOP-656: End to end submission engine


Copy link to this message
-
Review Request: SQOOP-656: End to end submission engine
Jarek Cecho 2012-10-31, 00:26

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/7783/
-----------------------------------------------------------

Review request for Sqoop.
Description
-------

This is sort of mega patch that is implementing first (not exactly stable and finished) version of submission engine. I've got like dozen other notes what to improve and implement, but the patch is extreme in size already. So I would like to propose getting it in, to give others chance to see what I'm working on at the moment and also retrieve their feedback. In meantime I'll continue my effort to make it very stable and very "nice" :-) My justification for doing that is that since entire sqoop 2 is alpha, it doesn't hurt committing alpha submission engine :-)

Submission engine is asynchronous pluggable interface for submitting mapreduce jobs. I've provided default mapreduce implementation (implementation that directly talks to JobTracker).

Procedure to get it working on dev box:
* Configure your hadoop cluster as usual
* Put config files to /etc/hadoop/conf or reconfigure sqoop to use different directory
* Put all required hadoop jars to tomcat lib folder (issue with resolving jar dependency is still open, so this is like temporal hack)
* Use client command submission to launch actual mapreduce jobs
This addresses bug SQOOP-656.
    https://issues.apache.org/jira/browse/SQOOP-656
Diffs
-----

  client/src/main/java/org/apache/sqoop/client/core/Environment.java cb92998d2054c8c7adbbf8fb1764fd5269d474b9
  client/src/main/java/org/apache/sqoop/client/core/RequestCache.java 03b1eb001f23aa5b3992a479ca9c2fc65136a0bd
  client/src/main/java/org/apache/sqoop/client/request/Request.java b243dfdfe4421236f865c50529a553314ff5b1b3
  client/src/main/java/org/apache/sqoop/client/request/SubmissionRequest.java PRE-CREATION
  client/src/main/java/org/apache/sqoop/client/shell/CreateConnectionFunction.java ea1e5d811da9fd817e20363f26498795b29985f8
  client/src/main/java/org/apache/sqoop/client/shell/SqoopShell.java c9ac848812ec2ae1ed4f1d15daff9eb6a0e19425
  client/src/main/java/org/apache/sqoop/client/shell/SubmissionCommand.java PRE-CREATION
  client/src/main/java/org/apache/sqoop/client/shell/SubmissionStartFunction.java PRE-CREATION
  client/src/main/java/org/apache/sqoop/client/shell/SubmissionStatusFunction.java PRE-CREATION
  client/src/main/java/org/apache/sqoop/client/shell/SubmissionStopFunction.java PRE-CREATION
  client/src/main/java/org/apache/sqoop/client/utils/FormFiller.java.orig d2fb6cecdf8a6baed02705bf17abf9a788b0c4c0
  client/src/main/java/org/apache/sqoop/client/utils/SubmissionDisplayer.java PRE-CREATION
  client/src/main/java/org/apache/sqoop/client/utils/ThrowableDisplayer.java 8d6e9b4814a53270e0a1920ce870ad0bef355e47
  common/src/main/java/org/apache/sqoop/common/ImmutableContext.java PRE-CREATION
  common/src/main/java/org/apache/sqoop/common/MapContext.java PRE-CREATION
  common/src/main/java/org/apache/sqoop/common/MutableContext.java PRE-CREATION
  common/src/main/java/org/apache/sqoop/common/MutableMapContext.java PRE-CREATION
  common/src/main/java/org/apache/sqoop/json/JobBean.java 7fd2ce52b7c563ad8570d8ebf88b2143ff5aea29
  common/src/main/java/org/apache/sqoop/json/SubmissionBean.java PRE-CREATION
  common/src/main/java/org/apache/sqoop/json/ThrowableBean.java 91914e84eee4c3e4ee8dc38a1449e51b54e2d2b2
  common/src/main/java/org/apache/sqoop/model/FormUtils.java 424810c1b36c4b8c5ebca143121a151e499d8231
  common/src/main/java/org/apache/sqoop/model/MSubmission.java PRE-CREATION
  common/src/main/java/org/apache/sqoop/submission/SubmissionStatus.java PRE-CREATION
  common/src/main/java/org/apache/sqoop/submission/counter/Counter.java PRE-CREATION
  common/src/main/java/org/apache/sqoop/submission/counter/CounterGroup.java PRE-CREATION
  common/src/main/java/org/apache/sqoop/submission/counter/Counters.java PRE-CREATION
  common/src/main/java/org/apache/sqoop/utils/ClassLoadingUtils.java 0c7d09f189c1a2fa96e9c5b782f6693e30cdd62a
  common/src/main/java/org/apache/sqoop/utils/ClassUtils.java PRE-CREATION
  common/src/test/java/org/apache/sqoop/json/TestSubmissionBean.java PRE-CREATION
  common/src/test/java/org/apache/sqoop/json/TestThrowableBean.java 19a0a27764f957998a7256c98126d2d0d29d1544
  common/src/test/java/org/apache/sqoop/utils/TestClassLoadingUtils.java e56e017e40bc21e84844ebb20ae72fc81c52eb92
  common/src/test/java/org/apache/sqoop/utils/TestClassUtils.java PRE-CREATION
  connector/connector-generic-jdbc/pom.xml be4cedd23210d9a5405d4f5ac5f4ee4fbeececf8
  connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcExecutor.java 702dd7e3ff50a203d7d5107de7178cc5517a29eb
  connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcExportDestroyer.java cba8f71f59df7068d6d043509c1cfc8927420aa8
  connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcExportInitializer.java a21dc7647e1b405510e050e8006435a293cd3b05
  connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcExportLoader.java 0289e29e041527f189a87c44e8e4737a27f83b95
  connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcImportDestroyer.java b0a0f62fc85c6284e346f68e7dfc62a05bb52623
  connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcImportExtractor.java 4499fda0e31ff4d01b30b8e39fefda9e37c2912f
  connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcImportInitializer.java 75f3e56a1077cda06c4b3d711c37a84ea7419c15
  connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcImportPartition.java cba313b43075713fb61c3f6661ceeac2c5a94b9e
  connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcImportPartitioner.java