Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Sqoop >> mail # user >> Sqoop 2 Import


Copy link to this message
-
Sqoop 2 Import
Hello,

I am migrating from Sqoop 1.4.3 to Sqoop 2, and am kind of stumbling my way
through the intro stuff. I'd like to import data from a MySQL database into
HDFS, and a simple import seems to be failing in 2, which works fine in 1.

- From (1.4.3) -
sqoop import --connect jdbc:mysql://127.0.0.1:3306/db --username kyle -P
--table kyle_table --target-dir /user/kyle/table

- To (1.99.2) -
sqoop:000> create job --xid 1 --type import
Creating job for connection with id 1
Please fill following values to create new job object
Name: test

Database configuration

Schema name:
Table name: kyle_table
Table SQL statement:
Table column names: *
Partition column name:
Boundary query:

Output configuration

Storage type:
  0 : HDFS
Choose: 0
Output format:
  0 : TEXT_FILE
  1 : SEQUENCE_FILE
Choose: 0
Output directory: /user/kyle/table2

Throttling resources

Extractors:
Loaders:
New job was successfully created with validation status FINE  and
persistent id 12
sqoop:000> submission start --jid 12
Submission details
Job id: 12
Status: BOOTING
Creation date: 2013-05-03 11:38:01 MST
Last update date: 2013-05-03 11:38:01 MST
External Id: job_1367275490217_0075
        http://server:8088/proxy/application_1367275490217_0075/
Progress: Progress is not available
sqoop:000> submission status --jid 12
Exception has occurred during processing command
Server has returned exception: Exception: java.lang.NullPointerException
Message:
--
On both cases, I see the jobs made their way to the job history, and on the
first, the output was saved to HDFS. On the second, I just have a blank
/user/kyle/table2 folder, and the task failed in Hadoop.
 - Hadoop Logs -
 2013-05-03 11:38:10,172 WARN [main] org.apache.hadoop.mapred.YarnChild:
Exception running child : java.lang.NullPointerException
 at java.lang.String.<init>(Unknown Source)
 at
org.apache.sqoop.job.mr.ConfigurationUtils.loadConfiguration(ConfigurationUtils.java:77)
 at
org.apache.sqoop.job.mr.ConfigurationUtils.getConnectorConnection(ConfigurationUtils.java:38)
 at org.apache.sqoop.job.mr.SqoopMapper.run(SqoopMapper.java:69)
 at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:726)
 at org.apache.hadoop.mapred.MapTask.run(MapTask.java:333)
 at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:157)
 at java.security.AccessController.doPrivileged(Native Method)
 at javax.security.auth.Subject.doAs(Unknown Source)
 at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1367)
 at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:152)

2013-05-03 11:38:10,172 ERROR [OutputFormatLoader-consumer]
org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor: Error while loading
data out of MR job.
java.lang.NullPointerException
 at java.lang.String.<init>(Unknown Source)
 at
org.apache.sqoop.job.mr.ConfigurationUtils.loadConfiguration(ConfigurationUtils.java:77)
 at
org.apache.sqoop.job.mr.ConfigurationUtils.getFrameworkConnection(ConfigurationUtils.java:50)
 at
org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor$ConsumerThread.run(SqoopOutputFormatLoadExecutor.java:204)
 at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
 at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
 at java.util.concurrent.FutureTask.run(Unknown Source)
 at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
 at java.lang.Thread.run(Unknown Source)
- sqoop.log -
2013-05-03 11:38:01,439 DEBUG repository.JdbcRepositoryTransaction
[org.apache.sqoop.repository.JdbcRepositoryTransaction.begin(JdbcRepositoryTransaction.java:73)]
Tx count-begin: 1, rollback: false
2013-05-03 11:38:01,452 DEBUG repository.JdbcRepositoryTransaction
[org.apache.sqoop.repository.JdbcRepositoryTransaction.commit(JdbcRepositoryTransaction.java:84)]
Tx count-commit: 1, rollback: false
2013-05-03 11:38:01,452 DEBUG repository.JdbcRepositoryTransaction
[org.apache.sqoop.repository.JdbcRepositoryTransaction.close(JdbcRepositoryTransaction.java:103)]
Tx count-close: 0, rollback: false
2013-05-03 11:38:01,452 INFO  repository.JdbcRepositoryTransaction
[org.apache.sqoop.repository.JdbcRepositoryTransaction.close(JdbcRepositoryTransaction.java:111)]
Attempting transaction commit
2013-05-03 11:38:01,453 DEBUG repository.JdbcRepositoryTransaction
[org.apache.sqoop.repository.JdbcRepositoryTransaction.begin(JdbcRepositoryTransaction.java:73)]
Tx count-begin: 1, rollback: false
2013-05-03 11:38:01,464 DEBUG repository.JdbcRepositoryTransaction
[org.apache.sqoop.repository.JdbcRepositoryTransaction.commit(JdbcRepositoryTransaction.java:84)]
Tx count-commit: 1, rollback: false
2013-05-03 11:38:01,464 DEBUG repository.JdbcRepositoryTransaction
[org.apache.sqoop.repository.JdbcRepositoryTransaction.close(JdbcRepositoryTransaction.java:103)]
Tx count-close: 0, rollback: false
2013-05-03 11:38:01,464 INFO  repository.JdbcRepositoryTransaction
[org.apache.sqoop.repository.JdbcRepositoryTransaction.close(JdbcRepositoryTransaction.java:111)]
Attempting transaction commit
2013-05-03 11:38:01,465 DEBUG framework.FrameworkManager
[org.apache.sqoop.framework.FrameworkManager.submit(FrameworkManager.java:417)]
Using callbacks:
Importer{initializer=org.apache.sqoop.connector.jdbc.GenericJdbcImportInitializer,
destroyer=org.apache.sqoop.connector.jdbc.GenericJdbcImportDestroyer,
partitioner=org.apache.sqoop.connector.jdbc.GenericJdbcImportPartitioner,
extractor=org.apache.sqoop.connector.jdbc.GenericJdbcImportExtractor}
2013-05-03 11:38:01,482 DEBUG jdbc.GenericJdbcImportInitializer
[org.apache.sqoop.connector.jdbc.GenericJdbcImportInitializer.configurePartitionProperties(GenericJdbcImportInitializer.java:154)]
Using minMaxQuery: SELECT MIN(id), MAX(id) FROM kyle_table
2013-05-03 11:38:01,483 INFO  jdbc.GenericJdbcImportInitializer
[org.apache.sqoop.connector.jdbc.GenericJdbcImportInitializer.configurePartitionProperties(GenericJdbcImportInitializer.java:169)]
Boundaries: min=1, m
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB