Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Sqoop >> mail # user >> Need help and tips for tthe following issue: No data get exported from hadoop to mysql using sqoop.


Copy link to this message
-
Need help and tips for tthe following issue: No data get exported from hadoop to mysql using sqoop.
Hi

I want to do the following: Export data stored in hadoop to MySql. It is
not working and I have been pulling my hair. I was hoping to get a bit of
help. Thank you in advance

The command is the following:

~/sqoop-1.4.2.bin__hadoop-1.0.0/bin/sqoop export --connect
jdbc:mysql://hostname:3306/analyticsdb --username username --password
password --table ml_ys_log_gmt_test --export-dir
hdfs:///mnt/var/lib/hadoop/dfs/logs_sanitized_test
--input-fields-terminated-by='\t'  --lines-terminated-by='\n' --verbose

On my mysqlserver in the database analyticsdb, I do have the following
table ml_ys_log_gmt_test

mysql> describe ml_ys_log_gmt_test;
+--------+-------------+------+-----+---------+-------+
| Field  | Type        | Null | Key | Default | Extra |
+--------+-------------+------+-----+---------+-------+
| mydate | varchar(32) | YES  |     | NULL    |       |
| mydata | varchar(32) | YES  |     | NULL    |       |
+--------+-------------+------+-----+---------+-------+

I can see the logs in hdfs

hadoop@ip-XX-XX-XX-XX:/mnt/var/lib/hadoop/steps/5$ hadoop dfs -ls
hdfs:///mnt/var/lib/hadoop/dfs/logs_sanitized_test
Found 2 items
drwxr-xr-x   - hadoop supergroup          0 2012-10-10 15:23
/mnt/var/lib/hadoop/dfs/logs_sanitized_test/dt=2012-10-01
drwxr-xr-x   - hadoop supergroup          0 2012-10-10 15:23
/mnt/var/lib/hadoop/dfs/logs_sanitized_test/dt=2012-10-02

and if i tail one of the file I see the correct data

hadoop@ip-XX-XX-XX-XX:/mnt/var/lib/hadoop/steps/5$ hadoop dfs -tail -f
hdfs:///mnt/var/lib/hadoop/dfs/logs_sanitized_test/dt=2012-10-01/part-m-00000
20121001230101 blablabla1
20121001230202 blablabla2
Here is the trace when I run the command. Please note that no data get
transferred. I would appreciate any tips. Thanks a lot!

Warning: /usr/lib/hbase does not exist! HBase imports will fail.
Please set $HBASE_HOME to the root of your HBase installation.
12/10/10 16:25:25 DEBUG tool.BaseSqoopTool: Enabled debug logging.
12/10/10 16:25:25 WARN tool.BaseSqoopTool: Setting your password on the
command-line is insecure. Consider using -P instead.
12/10/10 16:25:25 DEBUG sqoop.ConnFactory: Loaded manager factory:
com.cloudera.sqoop.manager.DefaultManagerFactory
12/10/10 16:25:25 DEBUG sqoop.ConnFactory: Trying ManagerFactory:
com.cloudera.sqoop.manager.DefaultManagerFactory
12/10/10 16:25:25 DEBUG manager.DefaultManagerFactory: Trying with scheme:
jdbc:mysql:
12/10/10 16:25:25 INFO manager.MySQLManager: Preparing to use a MySQL
streaming resultset.
12/10/10 16:25:25 DEBUG sqoop.ConnFactory: Instantiated ConnManager
org.apache.sqoop.manager.MySQLManager@5ef4f44a
12/10/10 16:25:25 INFO tool.CodeGenTool: Beginning code generation
12/10/10 16:25:25 DEBUG manager.SqlManager: No connection paramenters
specified. Using regular API for making connection.
12/10/10 16:25:26 DEBUG manager.SqlManager: Using fetchSize for next query:
-2147483648
12/10/10 16:25:26 INFO manager.SqlManager: Executing SQL statement: SELECT
t.* FROM `ml_ys_log_gmt_test` AS t LIMIT 1
12/10/10 16:25:26 DEBUG orm.ClassWriter: selected columns:
12/10/10 16:25:26 DEBUG orm.ClassWriter:   mydate
12/10/10 16:25:26 DEBUG orm.ClassWriter:   mydata
12/10/10 16:25:26 DEBUG manager.SqlManager: Using fetchSize for next query:
-2147483648
12/10/10 16:25:26 INFO manager.SqlManager: Executing SQL statement: SELECT
t.* FROM `ml_ys_log_gmt_test` AS t LIMIT 1
12/10/10 16:25:26 DEBUG orm.ClassWriter: Writing source file:
/tmp/sqoop-hadoop/compile/7f5cd67c0aa5dbf20256f72b30ae922b/ml_ys_log_gmt_test.java
12/10/10 16:25:26 DEBUG orm.ClassWriter: Table name: ml_ys_log_gmt_test
12/10/10 16:25:26 DEBUG orm.ClassWriter: Columns: mydate:12, mydata:12,
12/10/10 16:25:26 DEBUG orm.ClassWriter: sourceFilename is
ml_ys_log_gmt_test.java
12/10/10 16:25:26 DEBUG orm.CompilationManager: Found existing
/tmp/sqoop-hadoop/compile/7f5cd67c0aa5dbf20256f72b30ae922b/
12/10/10 16:25:26 INFO orm.CompilationManager: HADOOP_HOME is /home/hadoop
12/10/10 16:25:26 INFO orm.CompilationManager: Found hadoop core jar at:
/home/hadoop/hadoop-core.jar
12/10/10 16:25:26 DEBUG orm.CompilationManager: Adding source file:
/tmp/sqoop-hadoop/compile/7f5cd67c0aa5dbf20256f72b30ae922b/ml_ys_log_gmt_test.java
12/10/10 16:25:26 DEBUG orm.CompilationManager: Invoking javac with args:
12/10/10 16:25:26 DEBUG orm.CompilationManager:   -sourcepath
12/10/10 16:25:26 DEBUG orm.CompilationManager:
/tmp/sqoop-hadoop/compile/7f5cd67c0aa5dbf20256f72b30ae922b/
12/10/10 16:25:26 DEBUG orm.CompilationManager:   -d
12/10/10 16:25:26 DEBUG orm.CompilationManager:
/tmp/sqoop-hadoop/compile/7f5cd67c0aa5dbf20256f72b30ae922b/
12/10/10 16:25:26 DEBUG orm.CompilationManager:   -classpath
12/10/10 16:25:26 DEBUG orm.CompilationManager:
/home/hadoop/conf:/usr/lib/jvm/java-6-sun/lib/tools.jar:/home/hadoop:/home/hadoop/hadoop-core-1.0.3.jar:/home/hadoop/lib/activation-1.1.jar:/home/hadoop/lib/annotations.jar:/home/hadoop/lib/ant-1.8.1.jar:/home/hadoop/lib/ant-launcher-1.8.1.jar:/home/hadoop/lib/ant-nodeps-1.8.1.jar:/home/hadoop/lib/apache-jar-resource-bundle-1.4.jar:/home/hadoop/lib/asm-3.1.jar:/home/hadoop/lib/avro-1.5.3.jar:/home/hadoop/lib/avro-compiler-1.5.3.jar:/home/hadoop/lib/avro-ipc-1.5.3.jar:/home/hadoop/lib/avro-maven-plugin-1.5.3.jar:/home/hadoop/lib/aws-java-sdk-1.3.2.jar:/home/hadoop/lib/build-helper-maven-plugin-1.5.jar:/home/hadoop/lib/commons-beanutils-1.7.0.jar:/home/hadoop/lib/commons-beanutils-core-1.8.0.jar:/home/hadoop/lib/commons-cli-1.2.jar:/home/hadoop/lib/commons-codec-1.5.jar:/home/hadoop/lib/commons-collections-3.2.1.jar:/home/hadoop/lib/commons-configuration-1.6.jar:/home/hadoop/lib/commons-daemon-1.0.1.jar:/home/hadoop/lib/commons-digester-1.8.jar:/home/hadoop/lib/commons-el-1.0.jar:/home/hadoop/lib/commons-httpclient-3.1.jar:/home/hadoop/lib/commons-io-2.4.jar:/home/hadoop/lib/commons-lang-2.5.jar:/home/hadoop/lib/commons-logging-1.1.1.jar:/home/hadoop/lib/commons-logging-adapters-1.1.1.jar:/home/hadoop/lib/commons-logging-api-1.1.1.jar:/home/hadoop/lib/commons-math-2.1.jar:/ho