Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Sqoop >> mail # dev >> [jira] [Commented] (SQOOP-1056) Implement connection resiliency in Sqoop using pluggable failure handlers


Copy link to this message
-
[jira] [Commented] (SQOOP-1056) Implement connection resiliency in Sqoop using pluggable failure handlers

    [ https://issues.apache.org/jira/browse/SQOOP-1056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13888457#comment-13888457 ]

Hudson commented on SQOOP-1056:
-------------------------------

SUCCESS: Integrated in Sqoop-ant-jdk-1.6-hadoop100 #839 (See [https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop100/839/])
SQOOP-1056: Implement connection resiliency in Sqoop using pluggable failure handlers (venkat: https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=03fa9c53024671edb8807b3deb31b104c38a6a07)
* ivy/libraries.properties
* src/test/aop/build/aop.xml
* src/java/org/apache/sqoop/mapreduce/db/SQLFailureHandler.java
* src/java/org/apache/sqoop/manager/SqlManager.java
* src/java/org/apache/sqoop/mapreduce/sqlserver/SqlServerRecordReader.java
* src/java/org/apache/sqoop/mapreduce/SQLServerExportDBExecThread.java
* src/test/aop/org/apache/sqoop/mapreduce/SqlServerExportAspect.aj
* src/test/aop/org/apache/sqoop/fi/FiConfig.java
* src/java/org/apache/sqoop/mapreduce/SQLServerResilientUpdateOutputFormat.java
* src/java/org/apache/sqoop/mapreduce/SQLServerResilientExportOutputFormat.java
* src/java/org/apache/sqoop/mapreduce/SQLServerUpdateDBExecThread.java
* src/java/org/apache/sqoop/mapreduce/db/SQLServerDBInputFormat.java
* src/java/org/apache/sqoop/mapreduce/db/DBRecordReader.java
* src/java/org/apache/sqoop/mapreduce/SQLServerAsyncDBExecThread.java
* src/java/org/apache/sqoop/mapreduce/db/BasicRetrySQLFailureHandler.java
* build.xml
* src/java/org/apache/sqoop/manager/SQLServerManager.java
* ivy.xml
* src/java/org/apache/sqoop/mapreduce/db/SQLServerDBRecordReader.java
* src/test/aop/org/apache/sqoop/mapreduce/db/SqlServerImportAspect.aj
* src/test/aop/org/apache/sqoop/fi/ProbabilityModel.java
* src/java/org/apache/sqoop/mapreduce/db/DataDrivenDBRecordReader.java
* src/java/org/apache/sqoop/mapreduce/db/SQLServerConnectionFailureHandler.java
* src/test/fi-site.xml
> Implement connection resiliency in Sqoop using pluggable failure handlers
> -------------------------------------------------------------------------
>
>                 Key: SQOOP-1056
>                 URL: https://issues.apache.org/jira/browse/SQOOP-1056
>             Project: Sqoop
>          Issue Type: Improvement
>          Components: connectors/sqlserver
>            Reporter: Shuaishuai Nie
>            Assignee: Shuaishuai Nie
>             Fix For: 1.4.5
>
>         Attachments: SQOOP-1056-1057-combo.patch, SQOOP-1056.1.patch, SQOOP-1056.2.patch, SQOOP-1056.3.patch, Sqoop Connection Resiliency.docx
>
>
> Implement a pluggable way for handling connection failures, and/or intermittent errors in Sqoop. This is especially crucial in environments where the probability of connections getting reset or throttled is high.
> In case of intermittent failures in Sqoop, due to connection losses or server throttling, Sqoop does not recover from those failures. As a result, the running Sqoop task would eventually fail, and a new task is started. In those cases, Sqoop does not always guarantee that tasks can safely be restarted. For example, if part of the records is already committed to the database, then restarting the task would result in some failures like primary key violations. Even for Sqoop jobs which commit the records only at the end of the task, any failures towards the end of the task would involve reprocessing the whole range of split owned by the task, and any progress is lost.

--
This message was sent by Atlassian JIRA
(v6.1.5#6160)