Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Sqoop >> mail # user >> Fwd: sqoop incremental import fails- Violation of unique constraint SQOOP_SESSIONS_UNQ

Copy link to this message
Fwd: sqoop incremental import fails- Violation of unique constraint SQOOP_SESSIONS_UNQ
I find the following bug related to using incremental import and  direct
mode together.

  * [SQOOP-1078] - incremental import from database in direct mode
  * [SQOOP-976] - Incorrect SQL when incremental criteria is text column

Is there any syntax/data you can share about the correct syntax and
sequence of switches for using incremental import into hive using direct
mode import from mysql databases?

---------- Forwarded message ----------
From: Suhas Satish <[EMAIL PROTECTED]>
Date: Thu, Oct 10, 2013 at 1:15 PM
Subject: Re: sqoop incremental import fails- Violation of unique constraint

sqoop job --create signup_log --import --connect
jdbc:mysql://mydb/u1 --table signup_log --username u1
--password <password> --hive-import --hive-table signup_log
--incremental append --check-column sid --last-value 3276 --direct

What I notice is that sqoop is not doing an incremental import but trying
to do a full import from the beginning and fails because the map-reduce
output directory already exists on hadoop file system. Is there a bug in
sqoop command parsing of incremental import when the above parameters are
used together?

13/10/04 00:10:36 ERROR tool.ImportTool: Encountered IOException running
import job: org.apache.hadoop.mapred.FileAlreadyExistsException: Output
directory signup_log already exists
 at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:926)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:885)
 at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:885)
 at org.apache.hadoop.mapreduce.Job.submit(Job.java:573)
at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:603)
at org.apache.sqoop.mapreduce.ImportJobBase.runJob(ImportJobBase.java:151)
 at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:403)
at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:476)
 at org.apache.sqoop.tool.JobTool.execJob(JobTool.java:228)
at org.apache.sqoop.tool.JobTool.run(JobTool.java:283)
 at org.apache.sqoop.Sqoop.run(Sqoop.java:145)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
 at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:181)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:220)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:229)
 at org.apache.sqoop.Sqoop.main(Sqoop.java:238)

13/10/04 00:10:36 DEBUG hsqldb.HsqldbJobStorage: Flushing current
13/10/04 00:10:36 DEBUG hsqldb.HsqldbJobStorage: Closing connection


On Mon, Sep 30, 2013 at 10:42 AM, Jarek Jarcec Cecho <[EMAIL PROTECTED]>wrote:

> Hi Suhas,
> would you mind sharing with us Sqoop version, used Sqoop command line and
> entire log generated with parameter --verbose?
> Jarcec
> On Fri, Sep 27, 2013 at 02:19:39PM -0700, Suhas Satish wrote:
> > Hi Sqoop users,
> > Does anyone know whats the fix for this?
> >
> > I'm trying to  have incremental database changes imported into the
> > metastore that hive uses.
> >
> > 3/09/25 16:50:00 INFO hive.HiveImport: Time taken: 2.053 seconds
> > 13/09/25 16:50:00 INFO hive.HiveImport: Hive import complete.
> > 13/09/25 16:50:00 INFO hive.HiveImport: Export directory is empty,
> removing
> > it.
> > 13/09/25 16:50:00 INFO tool.ImportTool: Saving incremental import state
> to
> > the metastore
> > 13/09/25 16
Jarek Jarcec Cecho 2013-10-14, 15:04
Suhas Satish 2013-10-14, 16:04
Jarek Jarcec Cecho 2013-10-15, 17:08
Suhas Satish 2013-10-15, 18:38
Jarek Jarcec Cecho 2013-10-22, 16:10