Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Sqoop >> mail # dev >> Review Request 15589: SQOOP-1232 Sqoop2: Provide tooling infrastructure for Sqoop2


Copy link to this message
-
Re: Review Request 15589: SQOOP-1232 Sqoop2: Provide tooling infrastructure for Sqoop2


> On Nov. 27, 2013, 7:35 p.m., Abraham Elmahrek wrote:
> > Can you also rebase against sqoop2 master?

Thank you very much for the review Abe! Will rebase shortly.
> On Nov. 27, 2013, 7:35 p.m., Abraham Elmahrek wrote:
> > tools/src/main/java/org/apache/sqoop/tools/ToolRunner.java, lines 57-59
> > <https://reviews.apache.org/r/15589/diff/2/?file=391069#file391069line57>
> >
> >     Maybe throw an exception instead? This seems useful if we want to allow users to add custom tools. If not, then all the tools should be in BuiltinTools right?

I would prefer to give the administrators way how to execute custom tools if required.
> On Nov. 27, 2013, 7:35 p.m., Abraham Elmahrek wrote:
> > tools/src/main/java/org/apache/sqoop/tools/ToolRunner.java, line 56
> > <https://reviews.apache.org/r/15589/diff/2/?file=391069#file391069line56>
> >
> >     FQDN?? Do you mean full class name? ie a.b.c? canonical name might be better.

Good point, will update the comment. The canonical name won't work correctly for inner classes, so we need just the name.
- Jarek
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/15589/#review29499
-----------------------------------------------------------
On Nov. 26, 2013, 12:08 a.m., Jarek Cecho wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/15589/
> -----------------------------------------------------------
>
> (Updated Nov. 26, 2013, 12:08 a.m.)
>
>
> Review request for Sqoop.
>
>
> Bugs: SQOOP-1232
>     https://issues.apache.org/jira/browse/SQOOP-1232
>
>
> Repository: sqoop-sqoop2
>
>
> Description
> -------
>
> I've provided basic infrastructure for the tooling by adding new maven module sqoop-tools that is depending on the core module. I've also provided example tool called "verify" that will try to bootstrap all sqoop components to see if they are configured correctly.
>
> I was having difficulties to make run the code with the same environment as the Sqoop 2 server itself, but without the need to open the Tomcat for outside connections. I finally ended up using Tool mechanism that is exposed by Tomcat. Sadly this mechanism won't set up the environment exactly as we need it - common.loader and webapp class loaders are not loaded by this mechanism. so I've provided a class that will finish configuring the environment and call the usual ToolRunner. To keep the footprint of this class/jar file to a bare minimum I've placed it into standalone maven module that do not depend on any Sqoop libraries.
>
>
> Diffs
> -----
>
>   dist/pom.xml 9186a38b3b9c57a62731f01839a99dc7cad9fc99
>   dist/src/main/bin/sqoop.sh e3ed5ef4bbaba0c3048cb005d4e183e66768117e
>   pom.xml a4bc085ce808db912b6ca516b51c70529bd081e7
>   server/pom.xml a07ecf41c2aa687aa4a2a60367e1d1265610b7e9
>   tomcat/pom.xml PRE-CREATION
>   tomcat/src/main/java/org/apache/sqoop/tomcat/TomcatToolRunner.java PRE-CREATION
>   tools/pom.xml PRE-CREATION
>   tools/src/main/java/org/apache/sqoop/tools/Tool.java PRE-CREATION
>   tools/src/main/java/org/apache/sqoop/tools/ToolRunner.java PRE-CREATION
>   tools/src/main/java/org/apache/sqoop/tools/tool/BuiltinTools.java PRE-CREATION
>   tools/src/main/java/org/apache/sqoop/tools/tool/VerifyTool.java PRE-CREATION
>
> Diff: https://reviews.apache.org/r/15589/diff/
>
>
> Testing
> -------
>
> I'm currently not attaching any unit tests as this task should be better covered some sort of integration testing. I did however tested the provided verification tool on a real cluster both with correct configuration and miss configuration (missing Hadoop dependencies).
>
>
> Thanks,
>
> Jarek Cecho
>
>