Sqoop, mail # dev - Review Request: SQOOP-352 Export of avro data imported from database table (using sqoop import) fails on year

Review Request: SQOOP-352 Export of avro data imported from database table (using sqoop import) fails on year

Bilung Lee 2012-02-28, 21:40

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3845/
-----------------------------------------------------------

Review request for Sqoop, Tom White and Jarek Cecho.
Summary
-------

MySQL YEAR is mapped to java.sql.Date when importing to HDFS. On the way back an exception is thrown because the date and month fields are truncated (not used) - this is the MySQL behaviour.
This addresses bug SQOOP-352.
   https://issues.apache.org/jira/browse/SQOOP-352
Diffs
-----

  ./src/java/org/apache/sqoop/hive/TableDefWriter.java 1293040
  ./src/java/org/apache/sqoop/manager/ConnManager.java 1293040
  ./src/java/org/apache/sqoop/manager/MySQLManager.java 1293040
  ./src/java/org/apache/sqoop/manager/OracleManager.java 1293040
  ./src/java/org/apache/sqoop/manager/SqlManager.java 1293040
  ./src/java/org/apache/sqoop/mapreduce/JdbcExportJob.java 1293040
  ./src/java/org/apache/sqoop/orm/AvroSchemaGenerator.java 1293040
  ./src/java/org/apache/sqoop/orm/ClassWriter.java 1293040
  ./src/test/com/cloudera/sqoop/TestConnFactory.java 1293040
  ./src/test/com/cloudera/sqoop/manager/MySQLCompatTest.java 1293040

Diff: https://reviews.apache.org/r/3845/diff
Testing
-------
Thanks,

Bilung

Re: Review Request: SQOOP-352 Export of avro data imported from database table (using sqoop import) fails on year

Bilung Lee 2012-03-01, 05:17

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3845/
-----------------------------------------------------------

(Updated 2012-03-01 05:17:15.849853)
Review request for Sqoop, Arvind Prabhakar, Tom White, and Jarek Cecho.
Summary
-------

MySQL YEAR is mapped to java.sql.Date when importing to HDFS. On the way back an exception is thrown because the date and month fields are truncated (not used) - this is the MySQL behaviour.
This addresses bug SQOOP-352.
   https://issues.apache.org/jira/browse/SQOOP-352
Diffs
-----

  ./src/java/org/apache/sqoop/hive/TableDefWriter.java 1293040
  ./src/java/org/apache/sqoop/manager/ConnManager.java 1293040
  ./src/java/org/apache/sqoop/manager/MySQLManager.java 1293040
  ./src/java/org/apache/sqoop/manager/OracleManager.java 1293040
  ./src/java/org/apache/sqoop/manager/SqlManager.java 1293040
  ./src/java/org/apache/sqoop/mapreduce/JdbcExportJob.java 1293040
  ./src/java/org/apache/sqoop/orm/AvroSchemaGenerator.java 1293040
  ./src/java/org/apache/sqoop/orm/ClassWriter.java 1293040
  ./src/test/com/cloudera/sqoop/TestConnFactory.java 1293040
  ./src/test/com/cloudera/sqoop/manager/MySQLCompatTest.java 1293040

Diff: https://reviews.apache.org/r/3845/diff
Testing
-------
Thanks,

Bilung

Re: Review Request: SQOOP-352 Export of avro data imported from database table (using sqoop import) fails on year

Arvind Prabhakar 2012-03-02, 07:22

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3845/#review5549
-----------------------------------------------------------
Thanks for the patch Bilung. The logic of the change is sound but I am concerned about backward compatibility.
./src/java/org/apache/sqoop/manager/ConnManager.java
<https://reviews.apache.org/r/3845/#comment11992>

This change along with the change to toHiveType() and introduction of toAvroType() breaks backward compatibility.
One way to address this would be to keep the old abstract API of ConnManager and introduce new, non-abstract methods toJavaType(String columnName, int sqlType) which by default ignores the column name and returns toJavaType(sqlType). This way, you stay backwards compatible and built in connection managers can use the new functionality.

Another way that could possibly address this is by overwriting the getColumnTypesForRawQuery(String stmt) in MySQLManager and mapping the YEAR type to SMALLINT. Not sure if it will work or not, but is worth exploring.

- Arvind
On 2012-03-01 05:17:15, Bilung Lee wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/3845/
> -----------------------------------------------------------
>
> (Updated 2012-03-01 05:17:15)
>
>
> Review request for Sqoop, Arvind Prabhakar, Tom White, and Jarek Cecho.
>
>
> Summary
> -------
>
> MySQL YEAR is mapped to java.sql.Date when importing to HDFS. On the way back an exception is thrown because the date and month fields are truncated (not used) - this is the MySQL behaviour.
>
>
> This addresses bug SQOOP-352.
> https://issues.apache.org/jira/browse/SQOOP-352
>
>
> Diffs
> -----
>
> ./src/java/org/apache/sqoop/hive/TableDefWriter.java 1293040
> ./src/java/org/apache/sqoop/manager/ConnManager.java 1293040
> ./src/java/org/apache/sqoop/manager/MySQLManager.java 1293040
> ./src/java/org/apache/sqoop/manager/OracleManager.java 1293040
> ./src/java/org/apache/sqoop/manager/SqlManager.java 1293040
> ./src/java/org/apache/sqoop/mapreduce/JdbcExportJob.java 1293040
> ./src/java/org/apache/sqoop/orm/AvroSchemaGenerator.java 1293040
> ./src/java/org/apache/sqoop/orm/ClassWriter.java 1293040
> ./src/test/com/cloudera/sqoop/TestConnFactory.java 1293040
> ./src/test/com/cloudera/sqoop/manager/MySQLCompatTest.java 1293040
>
> Diff: https://reviews.apache.org/r/3845/diff
>
>
> Testing
> -------
>
>
> Thanks,
>
> Bilung
>
>

Re: Review Request: SQOOP-352 Export of avro data imported from database table (using sqoop import) fails on year

Bilung Lee 2012-03-02, 22:48

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3845/
-----------------------------------------------------------

(Updated 2012-03-02 22:48:24.704210)
Review request for Sqoop, Arvind Prabhakar, Tom White, and Jarek Cecho.
Changes
-------

Thanks for your comments, Arvind!

New patch is attached.
Summary
-------

MySQL YEAR is mapped to java.sql.Date when importing to HDFS. On the way back an exception is thrown because the date and month fields are truncated (not used) - this is the MySQL behaviour.
This addresses bug SQOOP-352.
   https://issues.apache.org/jira/browse/SQOOP-352
Diffs (updated)
-----

  ./src/java/org/apache/sqoop/hive/TableDefWriter.java 1296490
  ./src/java/org/apache/sqoop/manager/ConnManager.java 1296490
  ./src/java/org/apache/sqoop/manager/MySQLManager.java 1296490
  ./src/java/org/apache/sqoop/manager/OracleManager.java 1296490
  ./src/java/org/apache/sqoop/manager/SqlManager.java 1296490
  ./src/java/org/apache/sqoop/mapreduce/JdbcExportJob.java 1296490
  ./src/java/org/apache/sqoop/orm/AvroSchemaGenerator.java 1296490
  ./src/java/org/apache/sqoop/orm/ClassWriter.java 1296490
  ./src/test/com/cloudera/sqoop/manager/MySQLCompatTest.java 1296490

Diff: https://reviews.apache.org/r/3845/diff
Testing
-------
Thanks,

Bilung

Re: Review Request: SQOOP-352 Export of avro data imported from database table (using sqoop import) fails on year

Arvind Prabhakar 2012-03-06, 22:57

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3845/#review5661
-----------------------------------------------------------

Ship it!
+1

- Arvind
On 2012-03-02 22:48:24, Bilung Lee wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/3845/
> -----------------------------------------------------------
>
> (Updated 2012-03-02 22:48:24)
>
>
> Review request for Sqoop, Arvind Prabhakar, Tom White, and Jarek Cecho.
>
>
> Summary
> -------
>
> MySQL YEAR is mapped to java.sql.Date when importing to HDFS. On the way back an exception is thrown because the date and month fields are truncated (not used) - this is the MySQL behaviour.
>
>
> This addresses bug SQOOP-352.
> https://issues.apache.org/jira/browse/SQOOP-352
>
>
> Diffs
> -----
>
> ./src/java/org/apache/sqoop/hive/TableDefWriter.java 1296490
> ./src/java/org/apache/sqoop/manager/ConnManager.java 1296490
> ./src/java/org/apache/sqoop/manager/MySQLManager.java 1296490
> ./src/java/org/apache/sqoop/manager/OracleManager.java 1296490
> ./src/java/org/apache/sqoop/manager/SqlManager.java 1296490
> ./src/java/org/apache/sqoop/mapreduce/JdbcExportJob.java 1296490
> ./src/java/org/apache/sqoop/orm/AvroSchemaGenerator.java 1296490
> ./src/java/org/apache/sqoop/orm/ClassWriter.java 1296490
> ./src/test/com/cloudera/sqoop/manager/MySQLCompatTest.java 1296490
>
> Diff: https://reviews.apache.org/r/3845/diff
>
>
> Testing
> -------
>
>
> Thanks,
>
> Bilung
>
>