Hi all,

I'm trying to integrate our hadoop infrastructure with another team's
Informix database...

I have the JDBC drivers in place and test queries to most tables are fine
but we're hitting a couple of trip ups...

Some of the tables have grown in size and the primary key has been changed
to from serial to serial8 to cope with this.

When a query is run against such a table a java exception is raised with
the following stack trace:

java.io.IOException: SQLException in nextKeyValue
at org.apache.sqoop.mapreduce.db.DBRecordReader.nextKeyValue(DBRecordReader.java:265)
at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:458)
at org.apache.hadoop.mapreduce.task.MapContextImpl.nextKeyValue(MapContextImpl.java:76)
at org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.nextKeyValue(WrappedMapper.java:85)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:139)
at org.apache.sqoop.mapreduce.AutoProgressMapper.run(AutoProgressMapper.java:182)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:645)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:325)
at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332)
at org.apache.hadoop.mapred.Child.main(Child.java:262)
Has anyone encountered this or have any ideas how to work around it?
Kind regards,
James

Hi James,
I'm expecting that this exception is being raised on mapper side, would you mind sharing entire map task log with us?

Also, would you mind sharing your Hadoop and Sqoop version? Sqoop command line that you're using might be also helpful.

Jarcec

On Thu, Jan 17, 2013 at 11:00:16AM +0000, James Hogarth wrote:
> Hi all,
>
> I'm trying to integrate our hadoop infrastructure with another team's
> Informix database...
>
> I have the JDBC drivers in place and test queries to most tables are fine
> but we're hitting a couple of trip ups...
>
> Some of the tables have grown in size and the primary key has been changed
> to from serial to serial8 to cope with this.
>
> When a query is run against such a table a java exception is raised with
> the following stack trace:
>
> java.io.IOException: SQLException in nextKeyValue
> at org.apache.sqoop.mapreduce.db.DBRecordReader.nextKeyValue(DBRecordReader.java:265)
> at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:458)
> at org.apache.hadoop.mapreduce.task.MapContextImpl.nextKeyValue(MapContextImpl.java:76)
> at org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.nextKeyValue(WrappedMapper.java:85)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:139)
> at org.apache.sqoop.mapreduce.AutoProgressMapper.run(AutoProgressMapper.java:182)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:645)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:325)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332)
> at org.apache.hadoop.mapred.Child.main(Child.java:262)
>
>
> Has anyone encountered this or have any ideas how to work around it?
>
>
> Kind regards,
>
>
> James

> Hi James,
> I'm expecting that this exception is being raised on mapper side, would
> you mind sharing entire map task log with us?
>
> Also, would you mind sharing your Hadoop and Sqoop version? Sqoop command
> line that you're using might be also helpful.
>
>
We're using CDH4.1 (sqoop-1.4.1+54-1.cdh4.1.2.p0.21.el6.noarch)

 Attached is the log of the task attempt and anonymised copy/paste of the
console on the import attempt - you can see the command line on there.

I'm using a single mapper to try and avoid segmentation/boundary issues I
was bumping into as well...

On 17 January 2013 13:02, James Hogarth <[EMAIL PROTECTED]> wrote:

>
> Hi James,
>> I'm expecting that this exception is being raised on mapper side, would
>> you mind sharing entire map task log with us?
>>
>> Also, would you mind sharing your Hadoop and Sqoop version? Sqoop command
>> line that you're using might be also helpful.
>>
>>
> We're using CDH4.1 (sqoop-1.4.1+54-1.cdh4.1.2.p0.21.el6.noarch)
>
>  Attached is the log of the task attempt and anonymised copy/paste of the
> console on the import attempt - you can see the command line on there.
>
> I'm using a single mapper to try and avoid segmentation/boundary issues I
> was bumping into as well...
>
>
>

Hmm on further testing I'm not sure it is a serial8 issue actually and that
was a red herring...

When I restricted the dataset to something smaller the import went pretty
cleanly...

I'm trying to work out the boundary point where it starts getting issues at
the moment...

James

Hi James,
please accept my apologies for this late reply. I was travelling in last few days and I had limited access to my email.

I've checked task log attached to the previous email and it seems that the failure is actually somewhere in Informix JDBC driver:

2013-01-17 11:17:15,537 ERROR org.apache.sqoop.mapreduce.db.DBRecordReader: Top level exception:
java.sql.SQLException: System or internal error java.io.IOException
  at com.informix.util.IfxErrMsg.getSQLException(IfxErrMsg.java:488)
  at com.informix.jdbc.IfxSqli.a(IfxSqli.java:9532)
  at com.informix.jdbc.IfxSqli.receiveMessage(IfxSqli.java:2632)
  at com.informix.jdbc.IfxSqli.j(IfxSqli.java:2280)
  at com.informix.jdbc.IfxSqli.getaRow(IfxSqli.java:4681)
  at com.informix.jdbc.IfxResultSet.next(IfxResultSet.java:526)
  at org.apache.sqoop.mapreduce.db.DBRecordReader.nextKeyValue(DBRecordReader.java:238)
  at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:458)
  at org.apache.hadoop.mapreduce.task.MapContextImpl.nextKeyValue(MapContextImpl.java:76)
  at org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.nextKeyValue(WrappedMapper.java:85)
  at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:139)
  at org.apache.sqoop.mapreduce.AutoProgressMapper.run(AutoProgressMapper.java:182)
  at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:645)
  at org.apache.hadoop.mapred.MapTask.run(MapTask.java:325)
  at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
  at java.security.AccessController.doPrivileged(Native Method)
  at javax.security.auth.Subject.doAs(Subject.java:396)
  at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332)
  at org.apache.hadoop.mapred.Child.main(Child.java:262)
Caused by: java.io.IOException
  at com.informix.asf.IfxDataInputStream.readFully(IfxDataInputStream.java:148)
  at com.informix.asf.IfxDataInputStream.readSmallInt(IfxDataInputStream.java:453)
  at com.informix.jdbc.IfxSqli.receiveMessage(IfxSqli.java:2597)
  ... 16 more

I'm not familiar with Informix, so I'm not sure what might be causing such issues. Are there any logs on database side that might help understand what went wrong?

Jarcec

On Thu, Jan 17, 2013 at 04:48:49PM +0000, James Hogarth wrote:
> On 17 January 2013 13:02, James Hogarth <[EMAIL PROTECTED]> wrote:
>
> >
> > Hi James,
> >> I'm expecting that this exception is being raised on mapper side, would
> >> you mind sharing entire map task log with us?
> >>
> >> Also, would you mind sharing your Hadoop and Sqoop version? Sqoop command
> >> line that you're using might be also helpful.
> >>
> >>
> > We're using CDH4.1 (sqoop-1.4.1+54-1.cdh4.1.2.p0.21.el6.noarch)
> >
> >  Attached is the log of the task attempt and anonymised copy/paste of the
> > console on the import attempt - you can see the command line on there.
> >
> > I'm using a single mapper to try and avoid segmentation/boundary issues I
> > was bumping into as well...
> >
> >
> >
>
> Hmm on further testing I'm not sure it is a serial8 issue actually and that
> was a red herring...
>
> When I restricted the dataset to something smaller the import went pretty
> cleanly...
>
> I'm trying to work out the boundary point where it starts getting issues at
> the moment...
>
> James

On 22 January 2013 01:03, Jarek Jarcec Cecho <[EMAIL PROTECTED]> wrote:

> Hi James,
> please accept my apologies for this late reply. I was travelling in last
> few days and I had limited access to my email.
>
>
>
Hi Jarek,

Please accept my apologies for a delay in a response but work got the
better of me for a bit ;)

Chasing this down it ended up being (with near certainty) a DB issue...

If the dataset was too large it ended up exploding with the error
described....

When where clauses were put in place to cut things down (plus an increased
number of mappers) the sqoop would then complete...

I suspect it was the DB timing out the connection and consequently no
session for the query to complete on...

It's an extremely large database I'm importing on and a query for min/max
of the PK for a day can take up to around 3-4 minutes...