-
Re: Running Sqoop job from Oozie fails on create databaseJarek Jarcec Cecho 2012-11-27, 17:04
Hi John,
Sqoop is not supporting Hive integration when running from Oozie. Recommended workaround is to firstly run Sqoop import to temporary directory (no hive import) and than in separate Hive action load your data into Hive.
Jarcec
On Tue, Nov 27, 2012 at 04:49:59PM +0000, John Dasher wrote:
> Hi,
>
> I am attempting to run a sqoop job from oozie to load a table in Hive, incrementally. The oozie job errors with: "org.apache.hadoop.hive.ql.metadata.HiveException: javax.jdo.JDOFatalDataStoreException: Failed to create database '/var/lib/hive/metastore/metastore_db'"
>
> We have hive set up to store the meta-data in a MySql database. So I'm lost trying to find out where/why it's trying to create a database in Derby. Any pointers or information is greatly appreciated.
>
>
> Thank you,
>
> John
>
>
> We're using CDH4 (Free Edition):
>
> Hadoop 2.0.0-cdh4.0.1
>
> Oozie client build version: 3.1.3-cdh4.0.1
>
> Sqoop 1.4.1-cdh4.0.1
>
>
> Sqoop command and syslog below.
>
> Sqoop command arguments :
> job
> --meta-connect
> jdbc:hsqldb:hsql://hadoopdw4:16000/sqoop
> --exec
> sq_admin_users_hive
>
>
> syslog logs
>
>
>
> 2012-11-27 14:40:13,395 WARN mapreduce.Counters: Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead
> 2012-11-27 14:40:13,617 INFO org.apache.hadoop.util.NativeCodeLoader: Loaded the native-hadoop library
> 2012-11-27 14:40:14,048 WARN org.apache.hadoop.conf.Configuration: session.id is deprecated. Instead, use dfs.metrics.session-id
> 2012-11-27 14:40:14,049 INFO org.apache.hadoop.metrics.jvm.JvmMetrics: Initializing JVM Metrics with processName=MAP, sessionId> 2012-11-27 14:40:14,757 INFO org.apache.hadoop.util.ProcessTree: setsid exited with exit code 0
> 2012-11-27 14:40:14,763 INFO org.apache.hadoop.mapred.Task: Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@1ce3570c
> 2012-11-27 14:40:15,004 WARN org.apache.hadoop.io.compress.snappy.LoadSnappy: Snappy native library is available
> 2012-11-27 14:40:15,004 INFO org.apache.hadoop.io.compress.snappy.LoadSnappy: Snappy native library loaded
> 2012-11-27 14:40:15,011 WARN mapreduce.Counters: Counter name MAP_INPUT_BYTES is deprecated. Use FileInputFormatCounters as group name and BYTES_READ as counter name instead
> 2012-11-27 14:40:15,015 INFO org.apache.hadoop.mapred.MapTask: numReduceTasks: 0
> 2012-11-27 14:40:15,549 WARN org.apache.sqoop.tool.SqoopTool: $SQOOP_CONF_DIR has not been set in the environment. Cannot check for additional configuration.
> 2012-11-27 14:40:15,950 WARN org.apache.sqoop.ConnFactory: $SQOOP_CONF_DIR has not been set in the environment. Cannot check for additional configuration.
> 2012-11-27 14:40:16,036 INFO org.apache.sqoop.manager.MySQLManager: Preparing to use a MySQL streaming resultset.
> 2012-11-27 14:40:16,044 INFO org.apache.sqoop.tool.CodeGenTool: Beginning code generation
> 2012-11-27 14:40:16,598 INFO org.apache.sqoop.manager.SqlManager: Executing SQL statement: SELECT t.* FROM `admin_users` AS t LIMIT 1
> 2012-11-27 14:40:16,640 INFO org.apache.sqoop.manager.SqlManager: Executing SQL statement: SELECT t.* FROM `admin_users` AS t LIMIT 1
> 2012-11-27 14:40:16,660 INFO org.apache.sqoop.orm.CompilationManager: HADOOP_HOME is /usr/lib/hadoop-0.20-mapreduce
> 2012-11-27 14:40:16,661 INFO org.apache.sqoop.orm.CompilationManager: Found hadoop core jar at: /usr/lib/hadoop-0.20-mapreduce/hadoop-core.jar
> 2012-11-27 14:40:20,544 INFO org.apache.sqoop.orm.CompilationManager: Writing jar file: /tmp/sqoop-mapred/compile/7fef46c7a9af683cd26c7cf826f91b6e/admin_users.jar
> 2012-11-27 14:40:20,600 INFO org.apache.sqoop.tool.ImportTool: Incremental import based on column `updated_at`
> 2012-11-27 14:40:20,602 INFO org.apache.sqoop.tool.ImportTool: Lower bound value: '2012-11-26 21:12:01.0'
> 2012-11-27 14:40:20,602 INFO org.apache.sqoop.tool.ImportTool: Upper bound value: '2012-11-27 14:40:20.0'
Sqoop is not supporting Hive integration when running from Oozie. Recommended workaround is to firstly run Sqoop import to temporary directory (no hive import) and than in separate Hive action load your data into Hive.
Jarcec
On Tue, Nov 27, 2012 at 04:49:59PM +0000, John Dasher wrote:
> Hi,
>
> I am attempting to run a sqoop job from oozie to load a table in Hive, incrementally. The oozie job errors with: "org.apache.hadoop.hive.ql.metadata.HiveException: javax.jdo.JDOFatalDataStoreException: Failed to create database '/var/lib/hive/metastore/metastore_db'"
>
> We have hive set up to store the meta-data in a MySql database. So I'm lost trying to find out where/why it's trying to create a database in Derby. Any pointers or information is greatly appreciated.
>
>
> Thank you,
>
> John
>
>
> We're using CDH4 (Free Edition):
>
> Hadoop 2.0.0-cdh4.0.1
>
> Oozie client build version: 3.1.3-cdh4.0.1
>
> Sqoop 1.4.1-cdh4.0.1
>
>
> Sqoop command and syslog below.
>
> Sqoop command arguments :
> job
> --meta-connect
> jdbc:hsqldb:hsql://hadoopdw4:16000/sqoop
> --exec
> sq_admin_users_hive
>
>
> syslog logs
>
>
>
> 2012-11-27 14:40:13,395 WARN mapreduce.Counters: Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead
> 2012-11-27 14:40:13,617 INFO org.apache.hadoop.util.NativeCodeLoader: Loaded the native-hadoop library
> 2012-11-27 14:40:14,048 WARN org.apache.hadoop.conf.Configuration: session.id is deprecated. Instead, use dfs.metrics.session-id
> 2012-11-27 14:40:14,049 INFO org.apache.hadoop.metrics.jvm.JvmMetrics: Initializing JVM Metrics with processName=MAP, sessionId> 2012-11-27 14:40:14,757 INFO org.apache.hadoop.util.ProcessTree: setsid exited with exit code 0
> 2012-11-27 14:40:14,763 INFO org.apache.hadoop.mapred.Task: Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@1ce3570c
> 2012-11-27 14:40:15,004 WARN org.apache.hadoop.io.compress.snappy.LoadSnappy: Snappy native library is available
> 2012-11-27 14:40:15,004 INFO org.apache.hadoop.io.compress.snappy.LoadSnappy: Snappy native library loaded
> 2012-11-27 14:40:15,011 WARN mapreduce.Counters: Counter name MAP_INPUT_BYTES is deprecated. Use FileInputFormatCounters as group name and BYTES_READ as counter name instead
> 2012-11-27 14:40:15,015 INFO org.apache.hadoop.mapred.MapTask: numReduceTasks: 0
> 2012-11-27 14:40:15,549 WARN org.apache.sqoop.tool.SqoopTool: $SQOOP_CONF_DIR has not been set in the environment. Cannot check for additional configuration.
> 2012-11-27 14:40:15,950 WARN org.apache.sqoop.ConnFactory: $SQOOP_CONF_DIR has not been set in the environment. Cannot check for additional configuration.
> 2012-11-27 14:40:16,036 INFO org.apache.sqoop.manager.MySQLManager: Preparing to use a MySQL streaming resultset.
> 2012-11-27 14:40:16,044 INFO org.apache.sqoop.tool.CodeGenTool: Beginning code generation
> 2012-11-27 14:40:16,598 INFO org.apache.sqoop.manager.SqlManager: Executing SQL statement: SELECT t.* FROM `admin_users` AS t LIMIT 1
> 2012-11-27 14:40:16,640 INFO org.apache.sqoop.manager.SqlManager: Executing SQL statement: SELECT t.* FROM `admin_users` AS t LIMIT 1
> 2012-11-27 14:40:16,660 INFO org.apache.sqoop.orm.CompilationManager: HADOOP_HOME is /usr/lib/hadoop-0.20-mapreduce
> 2012-11-27 14:40:16,661 INFO org.apache.sqoop.orm.CompilationManager: Found hadoop core jar at: /usr/lib/hadoop-0.20-mapreduce/hadoop-core.jar
> 2012-11-27 14:40:20,544 INFO org.apache.sqoop.orm.CompilationManager: Writing jar file: /tmp/sqoop-mapred/compile/7fef46c7a9af683cd26c7cf826f91b6e/admin_users.jar
> 2012-11-27 14:40:20,600 INFO org.apache.sqoop.tool.ImportTool: Incremental import based on column `updated_at`
> 2012-11-27 14:40:20,602 INFO org.apache.sqoop.tool.ImportTool: Lower bound value: '2012-11-26 21:12:01.0'
> 2012-11-27 14:40:20,602 INFO org.apache.sqoop.tool.ImportTool: Upper bound value: '2012-11-27 14:40:20.0'
