Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Sqoop >> mail # user >> Sqoop not updating incremental.last.value when used with "--hive-import"


+
Mason 2012-12-27, 20:46
+
Jarek Jarcec Cecho 2012-12-28, 18:42
Copy link to this message
-
Re: Sqoop not updating incremental.last.value when used with "--hive-import"
hi Jarcec,

I dug deeper and it looks like a configuration error on my end
resulted in Hive throwing an error, which caused Sqoop to silently
abort the import, before updating incremental.last.value. I looked
around the source, but my Java is too rusty to figure out why Sqoop
didn't report failure after that error.

Still want a JIRA?

Logs are below.

### Sqoop CLI output ###
### ...a bunch of prior output... ###
12/12/27 10:57:38 INFO mapreduce.ImportJobBase: Transferred 9.1538 MB
in 49.4693 seconds (189.4804 KB/sec)
12/12/27 10:57:38 INFO mapreduce.ImportJobBase: Retrieved 41969 records.
12/12/27 10:57:38 INFO util.AppendUtils: Appending to directory dataset
12/12/27 10:57:39 INFO manager.SqlManager: Executing SQL statement:
SELECT t.* FROM `dataset` AS t LIMIT 1
12/12/27 10:57:40 WARN hive.TableDefWriter: Column created_at had to
be cast to a less precise type in Hive
12/12/27 10:57:40 WARN hive.TableDefWriter: Column updated_at had to
be cast to a less precise type in Hive
12/12/27 10:57:40 INFO hive.HiveImport: Removing temporary files from
import process: hdfs://localhost:9000/user/mason/dataset/_logs
12/12/27 10:57:40 INFO hive.HiveImport: Loading uploaded data into Hive
WARNING: org.apache.hadoop.metrics.jvm.EventCounter is deprecated.
Please use org.apache.hadoop.log.metrics.EventCounter in all the
log4j.properties files.
Logging initialized using configuration in
jar:file:/usr/local/Cellar/sqoop/1.4.2/libexec/lib/hive-common-0.9.0.jar!/hive-log4j.properties
Hive history file=/tmp/mason/hive_job_log_mason_201212271057_2014979494.txt
OK
Time taken: 4.503 seconds
Loading data to table default.dataset
OK
Time taken: 0.656 seconds
### end of output ###

### Hive error log ###
2012-12-27 10:57:40,138 WARN  conf.HiveConf
(HiveConf.java:<clinit>(70)) - hive-site.xml not found on CLASSPATH
2012-12-27 10:57:41,635 ERROR DataNucleus.Plugin
(Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core" requires
"org.eclipse.core.resources" but it cannot be resolved.
2012-12-27 10:57:41,635 ERROR DataNucleus.Plugin
(Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core" requires
"org.eclipse.core.resources" but it cannot be resolved.
2012-12-27 10:57:41,636 ERROR DataNucleus.Plugin
(Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core" requires
"org.eclipse.core.runtime" but it cannot be resolved.
2012-12-27 10:57:41,636 ERROR DataNucleus.Plugin
(Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core" requires
"org.eclipse.core.runtime" but it cannot be resolved.
2012-12-27 10:57:41,637 ERROR DataNucleus.Plugin
(Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core" requires
"org.eclipse.text" but it cannot be resolved.
2012-12-27 10:57:41,637 ERROR DataNucleus.Plugin
(Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core" requires
"org.eclipse.text" but it cannot be resolved.
2012-12-27 10:57:46,030 ERROR tool.ImportTool
(ImportTool.java:run(484)) - Encountered IOException running import
job: java.io.IOException: Exception thrown in Hive
at org.apache.sqoop.hive.HiveImport.executeScript(HiveImport.java:335)
at org.apache.sqoop.hive.HiveImport.importTable(HiveImport.java:226)
at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:415)
at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:476)
at org.apache.sqoop.tool.JobTool.execJob(JobTool.java:228)
at org.apache.sqoop.tool.JobTool.run(JobTool.java:283)
at org.apache.sqoop.Sqoop.run(Sqoop.java:145)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:181)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:220)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:229)
at org.apache.sqoop.Sqoop.main(Sqoop.java:238)
at com.cloudera.sqoop.Sqoop.main(Sqoop.java:57)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at org.apache.sqoop.hive.HiveImport.executeScript(HiveImport.java:308)
... 12 more
Caused by: ExitSecurityException
at org.apache.sqoop.util.SubprocessSecurityManager.checkExit(SubprocessSecurityManager.java:83)
at java.lang.Runtime.exit(Runtime.java:105)
at java.lang.System.exit(System.java:960)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:558)
... 17 more
### end of stack trace ###

On Fri, Dec 28, 2012 at 10:42 AM, Jarek Jarcec Cecho <[EMAIL PROTECTED]> wrote:
+
Jarek Jarcec Cecho 2013-01-01, 14:37