Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Sqoop >> mail # dev >> Review Request 15589: SQOOP-1232 Sqoop2: Provide tooling infrastructure for Sqoop2

Copy link to this message
Re: Review Request 15589: SQOOP-1232 Sqoop2: Provide tooling infrastructure for Sqoop2

This is an automatically generated e-mail. To reply, visit:

(Updated Dec. 2, 2013, 5:13 p.m.)
Review request for Sqoop.

Incorporated Abe's feedback.
Bugs: SQOOP-1232
Repository: sqoop-sqoop2

I've provided basic infrastructure for the tooling by adding new maven module sqoop-tools that is depending on the core module. I've also provided example tool called "verify" that will try to bootstrap all sqoop components to see if they are configured correctly.

I was having difficulties to make run the code with the same environment as the Sqoop 2 server itself, but without the need to open the Tomcat for outside connections. I finally ended up using Tool mechanism that is exposed by Tomcat. Sadly this mechanism won't set up the environment exactly as we need it - common.loader and webapp class loaders are not loaded by this mechanism. so I've provided a class that will finish configuring the environment and call the usual ToolRunner. To keep the footprint of this class/jar file to a bare minimum I've placed it into standalone maven module that do not depend on any Sqoop libraries.
Diffs (updated)

  dist/pom.xml 9186a38b3b9c57a62731f01839a99dc7cad9fc99
  dist/src/main/bin/sqoop.sh e3ed5ef4bbaba0c3048cb005d4e183e66768117e
  pom.xml a4bc085ce808db912b6ca516b51c70529bd081e7
  server/pom.xml a07ecf41c2aa687aa4a2a60367e1d1265610b7e9
  tomcat/pom.xml PRE-CREATION
  tomcat/src/main/java/org/apache/sqoop/tomcat/TomcatToolRunner.java PRE-CREATION
  tools/pom.xml PRE-CREATION
  tools/src/main/java/org/apache/sqoop/tools/Tool.java PRE-CREATION
  tools/src/main/java/org/apache/sqoop/tools/ToolRunner.java PRE-CREATION
  tools/src/main/java/org/apache/sqoop/tools/tool/BuiltinTools.java PRE-CREATION
  tools/src/main/java/org/apache/sqoop/tools/tool/VerifyTool.java PRE-CREATION

Diff: https://reviews.apache.org/r/15589/diff/

I'm currently not attaching any unit tests as this task should be better covered some sort of integration testing. I did however tested the provided verification tool on a real cluster both with correct configuration and miss configuration (missing Hadoop dependencies).

Jarek Cecho