Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Sqoop >> mail # dev >> Review Request: Fix for Sqoop-976: Incremental import with text column as criteria should fail early


+
Raghav Gautam 2013-05-29, 17:19
+
Jarek Cecho 2013-06-02, 15:43
+
Raghav Gautam 2013-06-03, 20:47
+
Jarek Cecho 2013-06-09, 15:49
+
Raghav Gautam 2013-06-18, 17:24
+
Raghav Gautam 2013-06-18, 17:58
+
Jarek Cecho 2013-06-19, 02:32
Copy link to this message
-
Re: Review Request: Fix for SQOOP-934: Potential connection timeout


> On June 19, 2013, 2:32 a.m., Jarek Cecho wrote:
> > src/java/org/apache/sqoop/hive/HiveImport.java, line 183
> > <https://reviews.apache.org/r/11940/diff/1/?file=307540#file307540line183>
> >
> >     Just brainstorming here, would it make sense to use Connection.isValid() [1] and discard the connection only in case that it's invalid? Rather than discarding even valid connection? (Such code would have to be in the ConnectionManager of course)
> >    
> >     Links:
> >     1: http://docs.oracle.com/javase/6/docs/api/java/sql/Connection.html#isValid(int)
>
> Venkat Ranganathan wrote:
>     Hi Jarek
>    
>     isValid is a new API added in JDK 6.   While we build with JDK6 as the base, there are many JDBC connectors still only built with JDK 5 JDBC support (most notably MySQL).  That precludes using this - one choice might be to use reflection for the availability of the method and use it and otherwise assume it is invalid

We've took a deep look into this with Venkat offline and discovered that MySQL JDBC Version 5.1.x is implementing JDBC4 compatible Connection object including the isValid() method. However the older version 5.0 do not implement this. As the 5.0 might still be used, I'm fine with discarding the connection and reconnecting at this point.
- Jarek
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/11940/#review22096
-----------------------------------------------------------
On June 18, 2013, 5:58 p.m., Raghav Gautam wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/11940/
> -----------------------------------------------------------
>
> (Updated June 18, 2013, 5:58 p.m.)
>
>
> Review request for Sqoop.
>
>
> Description
> -------
>
> This patches fixes the SQOOP-934. The connection created prior to map-reduce job can potentially timed out while the job is running. This will cause problems when this connection is reused for generating the hive script. This patch discards the connection prior to hive script generation. So, a fresh connection will be used for generating hive script.
>
>
> This addresses bug SQOOP-934.
>     https://issues.apache.org/jira/browse/SQOOP-934
>
>
> Diffs
> -----
>
>   src/java/org/apache/sqoop/hive/HiveImport.java 02596a6
>   src/java/org/apache/sqoop/manager/ConnManager.java 32e736c
>   src/java/org/apache/sqoop/manager/GenericJdbcManager.java 233e546
>   src/test/com/cloudera/sqoop/TestConnFactory.java c0b295e
>
> Diff: https://reviews.apache.org/r/11940/diff/
>
>
> Testing
> -------
>
> Unit tests pass.
> Manually tested hive import.
>
>
> Thanks,
>
> Raghav Gautam
>
>

+
Venkat Ranganathan 2013-06-19, 18:13
+
Raghav Gautam 2013-06-20, 20:09
+
Jarek Cecho 2013-06-20, 22:10