Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Sqoop, mail # dev - Review Request 12032: Incremental import using lastmodified mode always assumes column type to be timestamp, this patch fixes that


+
Raghav Gautam 2013-06-26, 13:21
+
Jarek Cecho 2013-06-26, 19:00
+
Raghav Gautam 2013-06-30, 14:18
Copy link to this message
-
Re: Review Request 12032: Incremental import using lastmodified mode always assumes column type to be timestamp, this patch fixes that
Jarek Cecho 2013-07-01, 16:07


> On June 26, 2013, 7 p.m., Jarek Cecho wrote:
> > Hi Raghav,
> > thank you very much for taking up this JIRA, appreciated! I'm wondering what is the reasoning for introducing a new query in the Oracle manager to retrive the check column type when the type has been already retrieved and seems to be stored in checkColumnType variable [1]? It seems to me that the real problem here is that the code in the ImportTool is simply overriding this type to Timestamp on [2] before passing it to the datetimeToQueryString [3]. Did you by any chance experiment with removing this line?
> >
> > It would be also great if you could include a new test for this, so that we can be sure that we won't break this in the future again.
> >
> > Jarcec
> >
> > Links:
> > 1: https://github.com/apache/sqoop/blob/trunk/src/java/org/apache/sqoop/tool/ImportTool.java#L70
> > 2: https://github.com/apache/sqoop/blob/trunk/src/java/org/apache/sqoop/tool/ImportTool.java#L286
> > 3: https://github.com/apache/sqoop/blob/trunk/src/java/org/apache/sqoop/manager/OracleManager.java#L578
>
> Raghav Gautam wrote:
>     Hi Jarek,
>     checkColumnType is a private variable which gets assigned in method getMaxColumnId.
>     https://github.com/apache/sqoop/blob/trunk/src/java/org/apache/sqoop/tool/ImportTool.java#L165
>     Since, this method does not get called in case incremental mode is lastmodified. My changes preserve existing behavior for the most part - changing it only for Oracle. Hence, I have added a new method to the OracleManager.
>    
>     I have added a test as per your comment.
>    
>     Thanks,
>     Raghav.

Thank you Raghav for the explanation, it make complete sense to me! I feel that introducing new method into the ConnManager class for retrieving one column type is a bit of overhaul when we can do simply manager.getColumnTypes(table, query).get(options.getIncrementalTestColumn()) [1] to achieve the same with existing code. Is there a particular reason why you've choose implementing a new method instead?

Links:
1: https://github.com/apache/sqoop/blob/trunk/src/java/org/apache/sqoop/manager/ConnManager.java#L342
- Jarek
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/12032/#review22413
-----------------------------------------------------------
On June 30, 2013, 2:18 p.m., Raghav Gautam wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/12032/
> -----------------------------------------------------------
>
> (Updated June 30, 2013, 2:18 p.m.)
>
>
> Review request for Sqoop.
>
>
> Bugs: SQOOP-906
>     https://issues.apache.org/jira/browse/SQOOP-906
>
>
> Repository: sqoop-trunk
>
>
> Description
> -------
>
> Incremental import using lastmodified mode always assumes column type to be TIMESTAMP which is causing issues with Oracle Connector. This patch fixes that.
>
>
> Diffs
> -----
>
>   src/java/org/apache/sqoop/manager/ConnManager.java c9e05da
>   src/java/org/apache/sqoop/manager/OracleManager.java edc888e
>   src/java/org/apache/sqoop/tool/ImportTool.java cb800b6
>   src/test/com/cloudera/sqoop/manager/OracleIncrementalImportTest.java PRE-CREATION
>   src/test/com/cloudera/sqoop/testutil/BaseSqoopTestCase.java 877d7f8
>
> Diff: https://reviews.apache.org/r/12032/diff/
>
>
> Testing
> -------
>
> Unit tests & oracle third party tests are passing. Also tested manually.
>
>
> Thanks,
>
> Raghav Gautam
>
>

+
Raghav Gautam 2013-06-30, 14:18
+
Raghav Gautam 2013-07-01, 20:02
+
Jarek Cecho 2013-07-09, 18:28
+
Raghav Gautam 2013-07-10, 18:33
+
Jarek Cecho 2013-07-11, 22:24
+
Raghav Gautam 2013-07-11, 22:52
+
Raghav Gautam 2013-07-12, 02:13
+
Jarek Cecho 2013-07-15, 17:21
+
Raghav Gautam 2013-07-15, 17:34