Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Sqoop >> mail # user >> Need help and tips for tthe following issue: No data get exported from hadoop to mysql using sqoop.


Copy link to this message
-
Need help and tips for tthe following issue: No data get exported from hadoop to mysql using sqoop.
Hi

I want to do the following: Export data stored in hadoop to MySql. It is
not working and I have been pulling my hair. I was hoping to get a bit of
help. Thank you in advance

The command is the following:

~/sqoop-1.4.2.bin__hadoop-1.0.0/bin/sqoop export --connect
jdbc:mysql://hostname:3306/analyticsdb --username username --password
password --table ml_ys_log_gmt_test --export-dir
hdfs:///mnt/var/lib/hadoop/dfs/logs_sanitized_test
--input-fields-terminated-by='\t'  --lines-terminated-by='\n' --verbose

On my mysqlserver in the database analyticsdb, I do have the following
table ml_ys_log_gmt_test

mysql> describe ml_ys_log_gmt_test;
+--------+-------------+------+-----+---------+-------+
| Field  | Type        | Null | Key | Default | Extra |
+--------+-------------+------+-----+---------+-------+
| mydate | varchar(32) | YES  |     | NULL    |       |
| mydata | varchar(32) | YES  |     | NULL    |       |
+--------+-------------+------+-----+---------+-------+

I can see the logs in hdfs

hadoop@ip-XX-XX-XX-XX:/mnt/var/lib/hadoop/steps/5$ hadoop dfs -ls
hdfs:///mnt/var/lib/hadoop/dfs/logs_sanitized_test
Found 2 items
drwxr-xr-x   - hadoop supergroup          0 2012-10-10 15:23
/mnt/var/lib/hadoop/dfs/logs_sanitized_test/dt=2012-10-01
drwxr-xr-x   - hadoop supergroup          0 2012-10-10 15:23
/mnt/var/lib/hadoop/dfs/logs_sanitized_test/dt=2012-10-02

and if i tail one of the file I see the correct data

hadoop@ip-XX-XX-XX-XX:/mnt/var/lib/hadoop/steps/5$ hadoop dfs -tail -f
hdfs:///mnt/var/lib/hadoop/dfs/logs_sanitized_test/dt=2012-10-01/part-m-00000
20121001230101 blablabla1
20121001230202 blablabla2
Here is the trace when I run the command. Please note that no data get
transferred. I would appreciate any tips. Thanks a lot!

Warning: /usr/lib/hbase does not exist! HBase imports will fail.
Please set $HBASE_HOME to the root of your HBase installation.
12/10/10 16:25:25 DEBUG tool.BaseSqoopTool: Enabled debug logging.
12/10/10 16:25:25 WARN tool.BaseSqoopTool: Setting your password on the
command-line is insecure. Consider using -P instead.
12/10/10 16:25:25 DEBUG sqoop.ConnFactory: Loaded manager factory:
com.cloudera.sqoop.manager.DefaultManagerFactory
12/10/10 16:25:25 DEBUG sqoop.ConnFactory: Trying ManagerFactory:
com.cloudera.sqoop.manager.DefaultManagerFactory
12/10/10 16:25:25 DEBUG manager.DefaultManagerFactory: Trying with scheme:
jdbc:mysql:
12/10/10 16:25:25 INFO manager.MySQLManager: Preparing to use a MySQL
streaming resultset.
12/10/10 16:25:25 DEBUG sqoop.ConnFactory: Instantiated ConnManager
org.apache.sqoop.manager.MySQLManager@5ef4f44a
12/10/10 16:25:25 INFO tool.CodeGenTool: Beginning code generation
12/10/10 16:25:25 DEBUG manager.SqlManager: No connection paramenters
specified. Using regular API for making connection.
12/10/10 16:25:26 DEBUG manager.SqlManager: Using fetchSize for next query:
-2147483648
12/10/10 16:25:26 INFO manager.SqlManager: Executing SQL statement: SELECT
t.* FROM `ml_ys_log_gmt_test` AS t LIMIT 1
12/10/10 16:25:26 DEBUG orm.ClassWriter: selected columns:
12/10/10 16:25:26 DEBUG orm.ClassWriter:   mydate
12/10/10 16:25:26 DEBUG orm.ClassWriter:   mydata
12/10/10 16:25:26 DEBUG manager.SqlManager: Using fetchSize for next query:
-2147483648
12/10/10 16:25:26 INFO manager.SqlManager: Executing SQL statement: SELECT
t.* FROM `ml_ys_log_gmt_test` AS t LIMIT 1
12/10/10 16:25:26 DEBUG orm.ClassWriter: Writing source file:
/tmp/sqoop-hadoop/compile/7f5cd67c0aa5dbf20256f72b30ae922b/ml_ys_log_gmt_test.java
12/10/10 16:25:26 DEBUG orm.ClassWriter: Table name: ml_ys_log_gmt_test
12/10/10 16:25:26 DEBUG orm.ClassWriter: Columns: mydate:12, mydata:12,
12/10/10 16:25:26 DEBUG orm.ClassWriter: sourceFilename is
ml_ys_log_gmt_test.java
12/10/10 16:25:26 DEBUG orm.CompilationManager: Found existing
/tmp/sqoop-hadoop/compile/7f5cd67c0aa5dbf20256f72b30ae922b/
12/10/10 16:25:26 INFO orm.CompilationManager: HADOOP_HOME is /home/hadoop
12/10/10 16:25:26 INFO orm.CompilationManager: Found hadoop core jar at:
/home/hadoop/hadoop-core.jar
12/10/10 16:25:26 DEBUG orm.CompilationManager: Adding source file:
/tmp/sqoop-hadoop/compile/7f5cd67c0aa5dbf20256f72b30ae922b/ml_ys_log_gmt_test.java
12/10/10 16:25:26 DEBUG orm.CompilationManager: Invoking javac with args:
12/10/10 16:25:26 DEBUG orm.CompilationManager:   -sourcepath
12/10/10 16:25:26 DEBUG orm.CompilationManager:
/tmp/sqoop-hadoop/compile/7f5cd67c0aa5dbf20256f72b30ae922b/
12/10/10 16:25:26 DEBUG orm.CompilationManager:   -d
12/10/10 16:25:26 DEBUG orm.CompilationManager:
/tmp/sqoop-hadoop/compile/7f5cd67c0aa5dbf20256f72b30ae922b/
12/10/10 16:25:26 DEBUG orm.CompilationManager:   -classpath
12/10/10 16:25:26 DEBUG orm.CompilationManager:
/home/hadoop/conf:/usr/lib/jvm/java-6-sun/lib/tools.jar:/home/hadoop:/home/hadoop/hadoop-core-1.0.3.jar:/home/hadoop/lib/activation-1.1.jar:/home/hadoop/lib/annotations.jar:/home/hadoop/lib/ant-1.8.1.jar:/home/hadoop/lib/ant-launcher-1.8.1.jar:/home/hadoop/lib/ant-nodeps-1.8.1.jar:/home/hadoop/lib/apache-jar-resource-bundle-1.4.jar:/home/hadoop/lib/asm-3.1.jar:/home/hadoop/lib/avro-1.5.3.jar:/home/hadoop/lib/avro-compiler-1.5.3.jar:/home/hadoop/lib/avro-ipc-1.5.3.jar:/home/hadoop/lib/avro-maven-plugin-1.5.3.jar:/home/hadoop/lib/aws-java-sdk-1.3.2.jar:/home/hadoop/lib/build-helper-maven-plugin-1.5.jar:/home/hadoop/lib/commons-beanutils-1.7.0.jar:/home/hadoop/lib/commons-beanutils-core-1.8.0.jar:/home/hadoop/lib/commons-cli-1.2.jar:/home/hadoop/lib/commons-codec-1.5.jar:/home/hadoop/lib/commons-collections-3.2.1.jar:/home/hadoop/lib/commons-configuration-1.6.jar:/home/hadoop/lib/commons-daemon-1.0.1.jar:/home/hadoop/lib/commons-digester-1.8.jar:/home/hadoop/lib/commons-el-1.0.jar:/home/hadoop/lib/commons-httpclient-3.1.jar:/home/hadoop/lib/commons-io-2.4.jar:/home/hadoop/lib/commons-lang-2.5.jar:/home/hadoop/lib/commons-logging-1.1.1.jar:/home/hadoop/lib/commons-logging-adapters-1.1.1.jar:/home/hadoop/lib/commons-logging-api-1.1.1.jar:/home/hadoop/lib/commons-math-2.1.jar:/ho
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB