Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Sqoop, mail # dev - Review Request: Fix for Sqoop-976: Incremental import with text column as criteria should fail early


Copy link to this message
-
Re: Review Request: Fix for SQOOP-934: Potential connection timeout
Venkat Ranganathan 2013-06-19, 18:13


> On June 19, 2013, 2:32 a.m., Jarek Cecho wrote:
> > src/java/org/apache/sqoop/hive/HiveImport.java, line 183
> > <https://reviews.apache.org/r/11940/diff/1/?file=307540#file307540line183>
> >
> >     Just brainstorming here, would it make sense to use Connection.isValid() [1] and discard the connection only in case that it's invalid? Rather than discarding even valid connection? (Such code would have to be in the ConnectionManager of course)
> >    
> >     Links:
> >     1: http://docs.oracle.com/javase/6/docs/api/java/sql/Connection.html#isValid(int)

Hi Jarek

isValid is a new API added in JDK 6.   While we build with JDK6 as the base, there are many JDBC connectors still only built with JDK 5 JDBC support (most notably MySQL).  That precludes using this - one choice might be to use reflection for the availability of the method and use it and otherwise assume it is invalid
> On June 19, 2013, 2:32 a.m., Jarek Cecho wrote:
> > src/java/org/apache/sqoop/manager/ConnManager.java, line 490
> > <https://reviews.apache.org/r/11940/diff/1/?file=307541#file307541line490>
> >
> >     I'm afraid that adding new abstract method to ConnManager is backward incompatible change. We should provide default implementation throwing NotImplementedException or something similar in the ConnManager class.

Good catch.  I agree this needs to be fixed
- Venkat
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/11940/#review22096
-----------------------------------------------------------
On June 18, 2013, 5:58 p.m., Raghav Gautam wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/11940/
> -----------------------------------------------------------
>
> (Updated June 18, 2013, 5:58 p.m.)
>
>
> Review request for Sqoop.
>
>
> Description
> -------
>
> This patches fixes the SQOOP-934. The connection created prior to map-reduce job can potentially timed out while the job is running. This will cause problems when this connection is reused for generating the hive script. This patch discards the connection prior to hive script generation. So, a fresh connection will be used for generating hive script.
>
>
> This addresses bug SQOOP-934.
>     https://issues.apache.org/jira/browse/SQOOP-934
>
>
> Diffs
> -----
>
>   src/java/org/apache/sqoop/hive/HiveImport.java 02596a6
>   src/java/org/apache/sqoop/manager/ConnManager.java 32e736c
>   src/java/org/apache/sqoop/manager/GenericJdbcManager.java 233e546
>   src/test/com/cloudera/sqoop/TestConnFactory.java c0b295e
>
> Diff: https://reviews.apache.org/r/11940/diff/
>
>
> Testing
> -------
>
> Unit tests pass.
> Manually tested hive import.
>
>
> Thanks,
>
> Raghav Gautam
>
>