Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Sqoop, mail # dev - Review Request 12949: Support custom postgres types (e.g. inet for IP addresses) - which includes postgres enums.


+
Nick White 2013-07-25, 15:38
Copy link to this message
-
Re: Review Request 12949: Support custom postgres types (e.g. inet for IP addresses) - which includes postgres enums.
Jarek Cecho 2013-07-28, 16:14

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/12949/#review24062
-----------------------------------------------------------
Hi Nick,
thank you very much for working on this patch. It seems that submitted patch is in addition to adding support for additional data types for PostgreSQL doing quite huge refactoring of the ConnManager class. As that refactoring is not backward compatible, I'm afraid that it can't be accepted.

Adding support for custom types should be fairy straighforward, you need to overload the toJavaType(), toHiveType() and toAvroType() method. For examples, take a look on the Oracle connector:

https://github.com/apache/sqoop/blob/trunk/src/java/org/apache/sqoop/manager/OracleManager.java#L551

Jarcec

- Jarek Cecho
On July 25, 2013, 3:38 p.m., Nick White wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/12949/
> -----------------------------------------------------------
>
> (Updated July 25, 2013, 3:38 p.m.)
>
>
> Review request for Sqoop.
>
>
> Repository: sqoop-trunk
>
>
> Description
> -------
>
> The patch adds a PostgresqlExportJob that replaces the OutputFormat (if needed) with a PostgresqlExportOutputFormat that inserts casts into the generated SQL statement (e.g. insert into mytable values (?, ?::inet, ?)). The patch also consolidates the various functions on ConnManager that return SQL type ints and type names into just one for each. This means the chunks of code in various parts of the codebase that select which of the three (former) methods to call can be replaced with a single call - and the call "routing" logic only appears in one place (ConnManager).
>
>
> Diffs
> -----
>
>   src/java/org/apache/sqoop/hive/TableDefWriter.java c9962e9
>   src/java/org/apache/sqoop/manager/ConnManager.java c84c859
>   src/java/org/apache/sqoop/manager/MySQLManager.java e1d5a36
>   src/java/org/apache/sqoop/manager/OracleManager.java 686bc19
>   src/java/org/apache/sqoop/manager/PostgresqlManager.java bd882b9
>   src/java/org/apache/sqoop/manager/SqlManager.java 2a4992d
>   src/java/org/apache/sqoop/mapreduce/ExportOutputFormat.java c2e39b1
>   src/java/org/apache/sqoop/mapreduce/JdbcExportJob.java fee78e0
>   src/java/org/apache/sqoop/mapreduce/PostgresqlExportJob.java PRE-CREATION
>   src/java/org/apache/sqoop/mapreduce/PostgresqlExportOutputFormat.java PRE-CREATION
>   src/java/org/apache/sqoop/mapreduce/hcat/SqoopHCatUtilities.java a109b40
>   src/java/org/apache/sqoop/orm/AvroSchemaGenerator.java 806bace
>   src/java/org/apache/sqoop/orm/ClassWriter.java 1bd2a41
>   src/test/com/cloudera/sqoop/manager/PostgresqlExportTest.java 0ac4599
>   src/test/com/cloudera/sqoop/manager/TestSqlManager.java 57855fa
>   src/test/org/apache/sqoop/manager/sqlserver/SQLServerManagerManualTest.java ee576c9
>
> Diff: https://reviews.apache.org/r/12949/diff/
>
>
> Testing
> -------
>
> I've added two cases to PostgresqlExportTest and tested them against a 9.3 database.
>
>
> Thanks,
>
> Nick White
>
>