Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Sqoop >> mail # user >> Sqoop not updating incremental.last.value when used with "--hive-import"


Copy link to this message
-
Re: Sqoop not updating incremental.last.value when used with "--hive-import"
hi Jarcec,

I dug deeper and it looks like a configuration error on my end
resulted in Hive throwing an error, which caused Sqoop to silently
abort the import, before updating incremental.last.value. I looked
around the source, but my Java is too rusty to figure out why Sqoop
didn't report failure after that error.

Still want a JIRA?

Logs are below.

### Sqoop CLI output ###
### ...a bunch of prior output... ###
12/12/27 10:57:38 INFO mapreduce.ImportJobBase: Transferred 9.1538 MB
in 49.4693 seconds (189.4804 KB/sec)
12/12/27 10:57:38 INFO mapreduce.ImportJobBase: Retrieved 41969 records.
12/12/27 10:57:38 INFO util.AppendUtils: Appending to directory dataset
12/12/27 10:57:39 INFO manager.SqlManager: Executing SQL statement:
SELECT t.* FROM `dataset` AS t LIMIT 1
12/12/27 10:57:40 WARN hive.TableDefWriter: Column created_at had to
be cast to a less precise type in Hive
12/12/27 10:57:40 WARN hive.TableDefWriter: Column updated_at had to
be cast to a less precise type in Hive
12/12/27 10:57:40 INFO hive.HiveImport: Removing temporary files from
import process: hdfs://localhost:9000/user/mason/dataset/_logs
12/12/27 10:57:40 INFO hive.HiveImport: Loading uploaded data into Hive
WARNING: org.apache.hadoop.metrics.jvm.EventCounter is deprecated.
Please use org.apache.hadoop.log.metrics.EventCounter in all the
log4j.properties files.
Logging initialized using configuration in
jar:file:/usr/local/Cellar/sqoop/1.4.2/libexec/lib/hive-common-0.9.0.jar!/hive-log4j.properties
Hive history file=/tmp/mason/hive_job_log_mason_201212271057_2014979494.txt
OK
Time taken: 4.503 seconds
Loading data to table default.dataset
OK
Time taken: 0.656 seconds
### end of output ###

### Hive error log ###
2012-12-27 10:57:40,138 WARN  conf.HiveConf
(HiveConf.java:<clinit>(70)) - hive-site.xml not found on CLASSPATH
2012-12-27 10:57:41,635 ERROR DataNucleus.Plugin
(Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core" requires
"org.eclipse.core.resources" but it cannot be resolved.
2012-12-27 10:57:41,635 ERROR DataNucleus.Plugin
(Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core" requires
"org.eclipse.core.resources" but it cannot be resolved.
2012-12-27 10:57:41,636 ERROR DataNucleus.Plugin
(Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core" requires
"org.eclipse.core.runtime" but it cannot be resolved.
2012-12-27 10:57:41,636 ERROR DataNucleus.Plugin
(Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core" requires
"org.eclipse.core.runtime" but it cannot be resolved.
2012-12-27 10:57:41,637 ERROR DataNucleus.Plugin
(Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core" requires
"org.eclipse.text" but it cannot be resolved.
2012-12-27 10:57:41,637 ERROR DataNucleus.Plugin
(Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core" requires
"org.eclipse.text" but it cannot be resolved.
2012-12-27 10:57:46,030 ERROR tool.ImportTool
(ImportTool.java:run(484)) - Encountered IOException running import
job: java.io.IOException: Exception thrown in Hive
at org.apache.sqoop.hive.HiveImport.executeScript(HiveImport.java:335)
at org.apache.sqoop.hive.HiveImport.importTable(HiveImport.java:226)
at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:415)
at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:476)
at org.apache.sqoop.tool.JobTool.execJob(JobTool.java:228)
at org.apache.sqoop.tool.JobTool.run(JobTool.java:283)
at org.apache.sqoop.Sqoop.run(Sqoop.java:145)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:181)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:220)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:229)
at org.apache.sqoop.Sqoop.main(Sqoop.java:238)
at com.cloudera.sqoop.Sqoop.main(Sqoop.java:57)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at org.apache.sqoop.hive.HiveImport.executeScript(HiveImport.java:308)
... 12 more
Caused by: ExitSecurityException
at org.apache.sqoop.util.SubprocessSecurityManager.checkExit(SubprocessSecurityManager.java:83)
at java.lang.Runtime.exit(Runtime.java:105)
at java.lang.System.exit(System.java:960)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:558)
... 17 more
### end of stack trace ###

On Fri, Dec 28, 2012 at 10:42 AM, Jarek Jarcec Cecho <[EMAIL PROTECTED]> wrote:
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB