Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Sqoop, mail # dev - Re: Review Request 12949: SQOOP-1149: Support custom postgres types (e.g. inet for IP addresses) - which includes postgres enums.


Copy link to this message
-
Re: Review Request 12949: SQOOP-1149: Support custom postgres types (e.g. inet for IP addresses) - which includes postgres enums.
Nick White 2013-07-28, 18:05

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/12949/
-----------------------------------------------------------

(Updated July 28, 2013, 6:05 p.m.)
Review request for Sqoop.
Changes
-------

I've revised the patch to maintain backwards compatibility. Maybe Sqoop 2.0 should hide how the types are gathered (e.g. from a table / query / stored procedure) from a database? The patch overrides toJavaType in PostgresqlManager as you suggest, but I still have to tweak the generated SQL when exporting as the postgres driver gives all custom types an SQL type of 1111 (OTHER). Postgres doesn't automatically convert Strings to custom types (e.g. http://stackoverflow.com/questions/3080470/jdbctemplate-and-inet-data-type); it needs explicit casts. Thanks -
Summary (updated)
-----------------

SQOOP-1149: Support custom postgres types (e.g. inet for IP addresses) - which includes postgres enums.
Repository: sqoop-trunk
Description
-------

The patch adds a PostgresqlExportJob that replaces the OutputFormat (if needed) with a PostgresqlExportOutputFormat that inserts casts into the generated SQL statement (e.g. insert into mytable values (?, ?::inet, ?)). The patch also consolidates the various functions on ConnManager that return SQL type ints and type names into just one for each. This means the chunks of code in various parts of the codebase that select which of the three (former) methods to call can be replaced with a single call - and the call "routing" logic only appears in one place (ConnManager).
Diffs (updated)
-----

  src/java/org/apache/sqoop/manager/ConnManager.java f4b22f9
  src/java/org/apache/sqoop/manager/PostgresqlManager.java bd882b9
  src/java/org/apache/sqoop/mapreduce/ExportOutputFormat.java c2e39b1
  src/java/org/apache/sqoop/mapreduce/PostgresqlExportJob.java PRE-CREATION
  src/java/org/apache/sqoop/mapreduce/PostgresqlExportOutputFormat.java PRE-CREATION
  src/test/com/cloudera/sqoop/manager/PostgresqlExportTest.java 0ac4599

Diff: https://reviews.apache.org/r/12949/diff/
Testing
-------

I've added two cases to PostgresqlExportTest and tested them against a 9.3 database.
Thanks,

Nick White

+
Nick White 2013-07-29, 01:37
+
Nick White 2013-08-01, 15:43
+
Nick White 2013-08-01, 15:47