Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Sqoop >> mail # user >> Sqoop and Teradata


Copy link to this message
-
Re: Sqoop and Teradata
The configuration property "sqoop.export.records.per.statement" can be used
to configure batch size in this case.

Have you tried Cloudera Connector for Teradata?  With the temporary tables
used by the connector, you may be able to get around the deadlock.
On Tue, Jan 24, 2012 at 5:04 PM, Srinivas Surasani <[EMAIL PROTECTED]> wrote:

> Hi All,
>
> I'm working on Hadoop CDH3 U0 and  Sqoop CDH3 U2.
>
> I'm trying to export csv files from HDFS to Teradata, it works well with
> setting mapper to "1" ( with batch loading of 1000 records at a time ).
> when I tried increasing the number of mappers to more than one I'm getting
> the following error. Also, is it possible to configure batch size at the
> time of export?
>
>
>  sqoop export  --verbose --driver com.teradata.jdbc.TeraDriver
> --connect jdbc:teradata://xxxx/database=xxxx  --username xxxxx --password
> xxxxx --table xxxx --export-dir /user/surasani/10minutes.txt
> --fields-terminated-by '|' -m 4 --batch
>
> 12/01/24 16:17:21 INFO mapred.JobClient:  map 3% reduce 0%
> 12/01/24 16:17:48 INFO mapred.JobClient: Task Id :
> attempt_201112211106_68553_m_000001_2, Status : FAILED
> *java.io.IOException: java.sql.BatchUpdateException: [Teradata Database]
> [TeraJDBC 13.00.00.07] [Error 2631] [SQLState 40001] Transaction ABORTed
> due to DeadLock*.
>         at
> com.cloudera.sqoop.mapreduce.AsyncSqlRecordWriter.write(AsyncSqlRecordWriter.java:223)
>         at
> com.cloudera.sqoop.mapreduce.AsyncSqlRecordWriter.write(AsyncSqlRecordWriter.java:49)
>         at
> org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.write(MapTask.java:530)
>         at
> org.apache.hadoop.mapreduce.TaskInputOutputContext.write(TaskInputOutputContext.java:80)
>         at
> com.cloudera.sqoop.mapreduce.TextExportMapper.map(TextExportMapper.java:82)
>         at
> com.cloudera.sqoop.mapreduce.TextExportMapper.map(TextExportMapper.java:40)
>         at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
>         at
> com.cloudera.sqoop.mapreduce.AutoProgressMapper.run(AutoProgressMapper.java:189)
>         at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:646)
>         at org.apache.hadoop.mapred.MapTask.run(MapTask.java:322)
>         at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at javax.security.auth.Subject.doAs(Subject.java:396)
>         at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1115)
>         at org.apache.hadoop.mapred.Child.main(Child.java:262)
> Caused by: java.sql.BatchUpdateException: [Teradata Database] [TeraJDBC
> 13.00.00.07] [Error 2631] [SQLState 40001] Transaction ABORTed due to
> DeadLock.
>         at
> com.teradata.jdbc.jdbc_4.util.ErrorFactory.convertToBatchUpdateException(ErrorFactory.java:116)
>         at
> com.teradata.jdbc.jdbc_4.PreparedStatement.executeBatchDMLArray(PreparedStatement.java:149)
>         at
> com.teradata.jdbc.jdbc_3.ifjdbc_4.TeraLocalPreparedStatement.executeBatch(TeraLocalPreparedStatement.java:299)
>         at
> com.cloudera.sqoop.mapreduce.AsyncSqlOutputFormat$AsyncSqlExecThread.run(AsyncSqlOutputFormat.java:232)
> Caused by: com.teradata.jdbc.jdbc_4.util.JDBCException: [Teradata
> Database] [TeraJDBC 13.00.00.07] [Error 2631] [SQLState 40001] Transaction
> ABORTed due to DeadLock.
>         at
> com.teradata.jdbc.jdbc_4.util.ErrorFactory.makeDatabaseSQLException(ErrorFactory.java:288)
>         at
> com.teradata.jdbc.jdbc_4.statemachine.ReceiveInitSubState.action(ReceiveInitSubState.java:102)
>         at
> com.teradata.jdbc.jdbc_4.statemachine.StatementReceiveState.subStateMachine(StatementReceiveState.java:285)
>         at
> com.teradata.jdbc.jdbc_4.statemachine.StatementReceiveState.action(StatementReceiveState.java:176)
>         at
> com.teradata.jdbc.jdbc_4.statemachine.StatementController.runBody(StatementController.java:108)
>         at
> com.teradata.jdbc.jdbc_4.statemachine.PreparedBatchStatementController.run(PreparedBatchStatementController.java:59)
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB