Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Sqoop, mail # dev - Review Request: SQOOP-656: End to end submission engine


Copy link to this message
-
Re: Review Request: SQOOP-656: End to end submission engine
Bilung Lee 2012-11-02, 21:58

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/7783/#review13067
-----------------------------------------------------------

Ship it!
Ship It!

- Bilung Lee
On Oct. 31, 2012, 12:26 a.m., Jarek Cecho wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/7783/
> -----------------------------------------------------------
>
> (Updated Oct. 31, 2012, 12:26 a.m.)
>
>
> Review request for Sqoop.
>
>
> Description
> -------
>
> This is sort of mega patch that is implementing first (not exactly stable and finished) version of submission engine. I've got like dozen other notes what to improve and implement, but the patch is extreme in size already. So I would like to propose getting it in, to give others chance to see what I'm working on at the moment and also retrieve their feedback. In meantime I'll continue my effort to make it very stable and very "nice" :-) My justification for doing that is that since entire sqoop 2 is alpha, it doesn't hurt committing alpha submission engine :-)
>
> Submission engine is asynchronous pluggable interface for submitting mapreduce jobs. I've provided default mapreduce implementation (implementation that directly talks to JobTracker).
>
> Procedure to get it working on dev box:
> * Configure your hadoop cluster as usual
> * Put config files to /etc/hadoop/conf or reconfigure sqoop to use different directory
> * Put all required hadoop jars to tomcat lib folder (issue with resolving jar dependency is still open, so this is like temporal hack)
> * Use client command submission to launch actual mapreduce jobs
>
>
> This addresses bug SQOOP-656.
>     https://issues.apache.org/jira/browse/SQOOP-656
>
>
> Diffs
> -----
>
>   client/src/main/java/org/apache/sqoop/client/core/Environment.java cb92998d2054c8c7adbbf8fb1764fd5269d474b9
>   client/src/main/java/org/apache/sqoop/client/core/RequestCache.java 03b1eb001f23aa5b3992a479ca9c2fc65136a0bd
>   client/src/main/java/org/apache/sqoop/client/request/Request.java b243dfdfe4421236f865c50529a553314ff5b1b3
>   client/src/main/java/org/apache/sqoop/client/request/SubmissionRequest.java PRE-CREATION
>   client/src/main/java/org/apache/sqoop/client/shell/CreateConnectionFunction.java ea1e5d811da9fd817e20363f26498795b29985f8
>   client/src/main/java/org/apache/sqoop/client/shell/SqoopShell.java c9ac848812ec2ae1ed4f1d15daff9eb6a0e19425
>   client/src/main/java/org/apache/sqoop/client/shell/SubmissionCommand.java PRE-CREATION
>   client/src/main/java/org/apache/sqoop/client/shell/SubmissionStartFunction.java PRE-CREATION
>   client/src/main/java/org/apache/sqoop/client/shell/SubmissionStatusFunction.java PRE-CREATION
>   client/src/main/java/org/apache/sqoop/client/shell/SubmissionStopFunction.java PRE-CREATION
>   client/src/main/java/org/apache/sqoop/client/utils/FormFiller.java.orig d2fb6cecdf8a6baed02705bf17abf9a788b0c4c0
>   client/src/main/java/org/apache/sqoop/client/utils/SubmissionDisplayer.java PRE-CREATION
>   client/src/main/java/org/apache/sqoop/client/utils/ThrowableDisplayer.java 8d6e9b4814a53270e0a1920ce870ad0bef355e47
>   common/src/main/java/org/apache/sqoop/common/ImmutableContext.java PRE-CREATION
>   common/src/main/java/org/apache/sqoop/common/MapContext.java PRE-CREATION
>   common/src/main/java/org/apache/sqoop/common/MutableContext.java PRE-CREATION
>   common/src/main/java/org/apache/sqoop/common/MutableMapContext.java PRE-CREATION
>   common/src/main/java/org/apache/sqoop/json/JobBean.java 7fd2ce52b7c563ad8570d8ebf88b2143ff5aea29
>   common/src/main/java/org/apache/sqoop/json/SubmissionBean.java PRE-CREATION
>   common/src/main/java/org/apache/sqoop/json/ThrowableBean.java 91914e84eee4c3e4ee8dc38a1449e51b54e2d2b2
>   common/src/main/java/org/apache/sqoop/model/FormUtils.java 424810c1b36c4b8c5ebca143121a151e499d8231