Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Sqoop >> mail # user >> Need help and tips for tthe following issue: No data get exported from hadoop to mysql using sqoop.


+
Matthieu Labour 2012-10-10, 16:30
Copy link to this message
-
Re: Need help and tips for tthe following issue: No data get exported from hadoop to mysql using sqoop.
Hi sir,
as far as I remember FileInputFormat is not doing recursive descent into subdirectories when looking for input files. Would you mind trying to export directory /mnt/var/lib/hadoop/dfs/logs_sanitized_test/dt=2012-10-01 to see if it will help? Something like

sqoop export ... --export-dir /mnt/var/lib/hadoop/dfs/logs_sanitized_test/dt=2012-10-01 ...

Jarcec

On Wed, Oct 10, 2012 at 12:30:56PM -0400, Matthieu Labour wrote:
> Hi
>
> I want to do the following: Export data stored in hadoop to MySql. It is
> not working and I have been pulling my hair. I was hoping to get a bit of
> help. Thank you in advance
>
> The command is the following:
>
> ~/sqoop-1.4.2.bin__hadoop-1.0.0/bin/sqoop export --connect
> jdbc:mysql://hostname:3306/analyticsdb --username username --password
> password --table ml_ys_log_gmt_test --export-dir
> hdfs:///mnt/var/lib/hadoop/dfs/logs_sanitized_test
> --input-fields-terminated-by='\t'  --lines-terminated-by='\n' --verbose
>
> On my mysqlserver in the database analyticsdb, I do have the following
> table ml_ys_log_gmt_test
>
> mysql> describe ml_ys_log_gmt_test;
> +--------+-------------+------+-----+---------+-------+
> | Field  | Type        | Null | Key | Default | Extra |
> +--------+-------------+------+-----+---------+-------+
> | mydate | varchar(32) | YES  |     | NULL    |       |
> | mydata | varchar(32) | YES  |     | NULL    |       |
> +--------+-------------+------+-----+---------+-------+
>
> I can see the logs in hdfs
>
> hadoop@ip-XX-XX-XX-XX:/mnt/var/lib/hadoop/steps/5$ hadoop dfs -ls
> hdfs:///mnt/var/lib/hadoop/dfs/logs_sanitized_test
> Found 2 items
> drwxr-xr-x   - hadoop supergroup          0 2012-10-10 15:23
> /mnt/var/lib/hadoop/dfs/logs_sanitized_test/dt=2012-10-01
> drwxr-xr-x   - hadoop supergroup          0 2012-10-10 15:23
> /mnt/var/lib/hadoop/dfs/logs_sanitized_test/dt=2012-10-02
>
> and if i tail one of the file I see the correct data
>
> hadoop@ip-XX-XX-XX-XX:/mnt/var/lib/hadoop/steps/5$ hadoop dfs -tail -f
> hdfs:///mnt/var/lib/hadoop/dfs/logs_sanitized_test/dt=2012-10-01/part-m-00000
> 20121001230101 blablabla1
> 20121001230202 blablabla2
>
>
> Here is the trace when I run the command. Please note that no data get
> transferred. I would appreciate any tips. Thanks a lot!
>
> Warning: /usr/lib/hbase does not exist! HBase imports will fail.
> Please set $HBASE_HOME to the root of your HBase installation.
> 12/10/10 16:25:25 DEBUG tool.BaseSqoopTool: Enabled debug logging.
> 12/10/10 16:25:25 WARN tool.BaseSqoopTool: Setting your password on the
> command-line is insecure. Consider using -P instead.
> 12/10/10 16:25:25 DEBUG sqoop.ConnFactory: Loaded manager factory:
> com.cloudera.sqoop.manager.DefaultManagerFactory
> 12/10/10 16:25:25 DEBUG sqoop.ConnFactory: Trying ManagerFactory:
> com.cloudera.sqoop.manager.DefaultManagerFactory
> 12/10/10 16:25:25 DEBUG manager.DefaultManagerFactory: Trying with scheme:
> jdbc:mysql:
> 12/10/10 16:25:25 INFO manager.MySQLManager: Preparing to use a MySQL
> streaming resultset.
> 12/10/10 16:25:25 DEBUG sqoop.ConnFactory: Instantiated ConnManager
> org.apache.sqoop.manager.MySQLManager@5ef4f44a
> 12/10/10 16:25:25 INFO tool.CodeGenTool: Beginning code generation
> 12/10/10 16:25:25 DEBUG manager.SqlManager: No connection paramenters
> specified. Using regular API for making connection.
> 12/10/10 16:25:26 DEBUG manager.SqlManager: Using fetchSize for next query:
> -2147483648
> 12/10/10 16:25:26 INFO manager.SqlManager: Executing SQL statement: SELECT
> t.* FROM `ml_ys_log_gmt_test` AS t LIMIT 1
> 12/10/10 16:25:26 DEBUG orm.ClassWriter: selected columns:
> 12/10/10 16:25:26 DEBUG orm.ClassWriter:   mydate
> 12/10/10 16:25:26 DEBUG orm.ClassWriter:   mydata
> 12/10/10 16:25:26 DEBUG manager.SqlManager: Using fetchSize for next query:
> -2147483648
> 12/10/10 16:25:26 INFO manager.SqlManager: Executing SQL statement: SELECT
> t.* FROM `ml_ys_log_gmt_test` AS t LIMIT 1
> 12/10/10 16:25:26 DEBUG orm.ClassWriter: Writing source file:
+
Matthieu Labour 2012-10-10, 17:16
+
Jarek Jarcec Cecho 2012-10-10, 17:27
+
Matthieu Labour 2012-10-10, 18:06
+
Matthieu Labour 2012-10-10, 21:22
+
Jarek Jarcec Cecho 2012-10-10, 23:58
+
Matthieu Labour 2012-10-11, 14:39
+
Jarek Jarcec Cecho 2012-10-11, 15:38
+
Jarek Jarcec Cecho 2012-10-15, 21:40
+
Matthieu Labour 2012-10-17, 21:53
+
Jarek Jarcec Cecho 2012-10-17, 21:58
+
Matthieu Labour 2012-10-17, 22:01