Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Sqoop, mail # user - Fwd: sqoop incremental import fails- Violation of unique constraint SQOOP_SESSIONS_UNQ


+
Suhas Satish 2013-10-11, 21:46
+
Jarek Jarcec Cecho 2013-10-14, 15:04
Copy link to this message
-
Re: Fwd: sqoop incremental import fails- Violation of unique constraint SQOOP_SESSIONS_UNQ
Suhas Satish 2013-10-14, 16:04
This is the concrete issue in sqoop 1.4.3 - The following command fails to
do an incremental import and tries to do a full import instead.

sqoop job --create signup_log --import --connect
jdbc:mysql://mydb/u1 --table signup_log --username u1
--password <password> --hive-import --hive-table signup_log
--incremental append --check-column sid --last-value 3276 --direct

Cheers,
Suhas.
On Mon, Oct 14, 2013 at 8:04 AM, Jarek Jarcec Cecho <[EMAIL PROTECTED]>wrote:

> Hi Suhas,
> to my best knowledge, direct import from MySQL to Hive should be working
> in incremental mode. Is there any concrete issue that you are facing?
>
> Jarcec
>
> On Fri, Oct 11, 2013 at 02:46:10PM -0700, Suhas Satish wrote:
> > I find the following bug related to using incremental import and  direct
> > mode together.
> >
> >   * [SQOOP-1078] - incremental import from database in direct mode
> >   * [SQOOP-976] - Incorrect SQL when incremental criteria is text column
> >
> > Is there any syntax/data you can share about the correct syntax and
> > sequence of switches for using incremental import into hive using direct
> > mode import from mysql databases?
> >
> > Thanks,
> > Suhas.
> >
> >
> > ---------- Forwarded message ----------
> > From: Suhas Satish <[EMAIL PROTECTED]>
> > Date: Thu, Oct 10, 2013 at 1:15 PM
> > Subject: Re: sqoop incremental import fails- Violation of unique
> constraint
> > SQOOP_SESSIONS_UNQ
> > To: user <[EMAIL PROTECTED]>
> >
> >
> > sqoop1.4.3
> >
> > sqoop job --create signup_log --import --connect
> > jdbc:mysql://mydb/u1 --table signup_log --username u1
> > --password <password> --hive-import --hive-table signup_log
> > --incremental append --check-column sid --last-value 3276 --direct
> >
> > What I notice is that sqoop is not doing an incremental import but trying
> > to do a full import from the beginning and fails because the map-reduce
> > output directory already exists on hadoop file system. Is there a bug in
> > sqoop command parsing of incremental import when the above parameters are
> > used together?
> >
> > 13/10/04 00:10:36 ERROR tool.ImportTool: Encountered IOException running
> > import job: org.apache.hadoop.mapred.FileAlreadyExistsException: Output
> > directory signup_log already exists
> > at
> >
> org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.checkOutputSpecs(FileOutputFormat.java:134)
> >  at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:926)
> > at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:885)
> >  at java.security.AccessController.doPrivileged(Native Method)
> > at javax.security.auth.Subject.doAs(Subject.java:415)
> >  at
> >
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1127)
> > at
> org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:885)
> >  at org.apache.hadoop.mapreduce.Job.submit(Job.java:573)
> > at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:603)
> >  at
> >
> org.apache.sqoop.mapreduce.ImportJobBase.doSubmitJob(ImportJobBase.java:173)
> > at
> org.apache.sqoop.mapreduce.ImportJobBase.runJob(ImportJobBase.java:151)
> >  at
> >
> org.apache.sqoop.mapreduce.ImportJobBase.runImport(ImportJobBase.java:221)
> > at
> >
> org.apache.sqoop.manager.DirectMySQLManager.importTable(DirectMySQLManager.java:92)
> >  at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:403)
> > at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:476)
> >  at org.apache.sqoop.tool.JobTool.execJob(JobTool.java:228)
> > at org.apache.sqoop.tool.JobTool.run(JobTool.java:283)
> >  at org.apache.sqoop.Sqoop.run(Sqoop.java:145)
> > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
> >  at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:181)
> > at org.apache.sqoop.Sqoop.runTool(Sqoop.java:220)
> > at org.apache.sqoop.Sqoop.runTool(Sqoop.java:229)
> >  at org.apache.sqoop.Sqoop.main(Sqoop.java:238)
> >
> > 13/10/04 00:10:36 DEBUG hsqldb.HsqldbJobStorage: Flushing current
> > transaction
> > 13/10/04 00:10
+
Jarek Jarcec Cecho 2013-10-15, 17:08
+
Suhas Satish 2013-10-15, 18:38
+
Jarek Jarcec Cecho 2013-10-22, 16:10