Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Sqoop, mail # dev - Review Request: SQOOP-749: Exports Using Stored Procedures (Functions)


Copy link to this message
-
Re: Review Request: SQOOP-749: Exports Using Stored Procedures (Functions)
Jarek Cecho 2013-01-06, 11:36

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/8806/#review15101
-----------------------------------------------------------
Hi Nick,
thank you very much for your hard work on this issue. The high level functionality looks good to me. Would you mind providing also documentation for user? (src/doc/user)

I'm afraid that we can't upgrade to HSQLDB 2.0 as it would break internal metastore. As an alternative we might use derby or create tests in third party "module" (e.g separate tests for MySQL, PostgreSQL, Microsoft SQL Server, Oracle, ...).

I would personally really like to see the stored procedure tests for majority of third party tests to be sure that it will work correctly on given databases.

Jarcec

- Jarek Cecho
On Jan. 3, 2013, 12:36 a.m., Nick White wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/8806/
> -----------------------------------------------------------
>
> (Updated Jan. 3, 2013, 12:36 a.m.)
>
>
> Review request for Sqoop and Jarek Cecho.
>
>
> Description
> -------
>
> It'd be useful if you could use stored procedures (or functions) to insert data - currently you can only use insert or update statements (or upsert / merges, depending on the SqlManager you're using). This would help sqoop adoption / migration into environments which have existing, SQL-based data import workflows.
>
> The attached patch adds a --call argument to the export tool that can be used in place of --table.
>
> Note that this patch depends on HSQLDB 2.x, whereas trunk currently depends on 1.8, so it can't be committed until that situation's resolved (see SQOOP-519 and linked tickets).
>
>
> This addresses bug SQOOP-749.
>     https://issues.apache.org/jira/browse/SQOOP-749
>
>
> Diffs
> -----
>
>   src/java/org/apache/sqoop/SqoopOptions.java 613f797
>   src/java/org/apache/sqoop/manager/ConnManager.java 21eea93
>   src/java/org/apache/sqoop/manager/SqlManager.java 3a52c6d
>   src/java/org/apache/sqoop/mapreduce/ExportCallOutputFormat.java PRE-CREATION
>   src/java/org/apache/sqoop/mapreduce/JdbcCallExportJob.java PRE-CREATION
>   src/java/org/apache/sqoop/mapreduce/JdbcExportJob.java 7c52110
>   src/java/org/apache/sqoop/orm/ClassWriter.java b73711e
>   src/java/org/apache/sqoop/tool/BaseSqoopTool.java d795646
>   src/java/org/apache/sqoop/tool/ExportTool.java acd296d
>   src/test/com/cloudera/sqoop/SmokeTests.java 76df6cf
>   src/test/com/cloudera/sqoop/TestConnFactory.java 893b388
>   src/test/com/cloudera/sqoop/TestExport.java eba10aa
>   src/test/com/cloudera/sqoop/testutil/ExportJobTestCase.java 4f6fd37
>   src/test/org/apache/sqoop/TestExportUsingProcedure.java PRE-CREATION
>
> Diff: https://reviews.apache.org/r/8806/diff/
>
>
> Testing
> -------
>
> I've added a subclass of TestExport to make sure all the usual export functionality works. I've also used it on live data exporting to postgres.
>
>
> Thanks,
>
> Nick White
>
>