Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Sqoop >> mail # user >> Intermittent problems with sqoop using Oracle JDBC driver


Copy link to this message
-
Re: Intermittent problems with sqoop using Oracle JDBC driver
Hi Andre,
contributions are greatly appreciated! Just open a JIRA [1], enhance the troubleshooting guide [2] and attach the patch!

Jarcec

Links:
1: https://issues.apache.org/jira/browse/SQOOP
2: https://github.com/apache/sqoop/blob/trunk/src/docs/user/troubleshooting.txt

On Tue, Jul 16, 2013 at 06:47:46AM +1000, Andre Araujo wrote:
> Hi, Jarek,
>
> I believe it certainly would. Is that something I can help with?
>
> Andre
>
>
>
> On 16 July 2013 02:55, Jarek Jarcec Cecho <[EMAIL PROTECTED]> wrote:
>
> > Hi Andre,
> > thank you for the blog post. Do you think that it would be helpful to put
> > the additional information into the Sqoop Troubleshooting guide?
> >
> >
> > http://sqoop.apache.org/docs/1.4.3/SqoopUserGuide.html#_oracle_connection_reset_errors
> >
> > Jarcec
> >
> > On Mon, Jul 15, 2013 at 01:12:27PM +1000, Andre Araujo wrote:
> > > Thanks, David.
> > >
> > > My blog post is pending revision and should be published soon. I'll post
> > > the final link when it does.
> > > For the time being, please see below a copy of it without the formatting.
> > > What worked for me was a combination of two things:
> > >
> > >    - passing the "-D
> > >    mapred.child.java.opts="-Djava.security.egd=file:/dev/../dev/urandom""
> > >    parameter to sqoop
> > >    - setting the java.security.egd parameter in the HADOOP_OPTS variable,
> > >    so that it was passed to "${HADOOP_COMMON_HOME}/bin/hadoop
> > >     org.apache.sqoop.Sqoop"
> > >
> > > Regards,
> > > Andre
> > >
> > > -----------------------------------
> > >
> > > I’ve been using Sqoop to load data into HDFS from Oracle. I’m using
> > version
> > > 1.4.3 of Sqoop, running on a Linux machine and using the Oracle JDBC
> > driver
> > > with JDK 1.6.
> > >
> > > I was getting intermittent connection resets when trying to import data.
> > > After much troubleshooting, I eventually found the problem to be related
> > to
> > > a known issue with the JDBC driver and found a way to work around it,
> > which
> > > is described in the post
> > >
> > >
> > > The problem
> > >
> > > I noticed that when I was importing data at times where the machine I was
> > > running the sqoop client at was mostly idle, everything would run just
> > > fine. However, at times when others started to work on the same machine
> > and
> > > it became a bit busier, I would start to get the errors below
> > > intermittently:
> > >
> > > [araujo@client ~]$ sqoop import --connect jdbc:oracle:thin:user/pwd@host
> > /orcl
> > > -m 1 --query 'select 1 from dual where $CONDITIONS' --target-dir test
> > > 13/07/12 09:35:39 INFO manager.SqlManager: Using default fetchSize of
> > 1000
> > > 13/07/12 09:35:39 INFO tool.CodeGenTool: Beginning code generation
> > > 13/07/12 09:37:53 ERROR manager.SqlManager: Error executing statement:
> > > java.sql.SQLRecoverableException: IO Error: Connection reset
> > >  at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:467)
> > > at oracle.jdbc.driver.PhysicalConnection.(PhysicalConnection.java:546)
> > >         ...
> > > Caused by: java.net.SocketException: Connection reset
> > > at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:96)
> > > at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
> > >  ... 24 more
> > > After some troubleshooting and googling, I found that the problem seemed
> > to
> > > be related to the issue described in the following articles:
> > >
> > >
> > http://stackoverflow.com/questions/2327220/oracle-jdbc-intermittent-connection-issue/
> > > https://forums.oracle.com/message/3701989/
> > >
> > > Confirming the problem
> > >
> > > To ensure the problem was the same as the one described in the articles,
> > > and not something else intrinsic to Sqoop, I created a small Java program
> > > that simply connected to the database. I was able to reproduce the issue
> > > using it:
> > >
> > > [araujo@client TestConn]$ time java TestConn
> > > Exception in thread "main" java.sql.SQLRecoverableException: IO Error:
> > > Connection reset