Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Sqoop >> mail # user >> Need help and tips for tthe following issue: No data get exported from hadoop to mysql using sqoop.


Copy link to this message
-
Re: Need help and tips for tthe following issue: No data get exported from hadoop to mysql using sqoop.
Hi sir,
as far as I remember FileInputFormat is not doing recursive descent into subdirectories when looking for input files. Would you mind trying to export directory /mnt/var/lib/hadoop/dfs/logs_sanitized_test/dt=2012-10-01 to see if it will help? Something like

sqoop export ... --export-dir /mnt/var/lib/hadoop/dfs/logs_sanitized_test/dt=2012-10-01 ...

Jarcec

On Wed, Oct 10, 2012 at 12:30:56PM -0400, Matthieu Labour wrote:
> Hi
>
> I want to do the following: Export data stored in hadoop to MySql. It is
> not working and I have been pulling my hair. I was hoping to get a bit of
> help. Thank you in advance
>
> The command is the following:
>
> ~/sqoop-1.4.2.bin__hadoop-1.0.0/bin/sqoop export --connect
> jdbc:mysql://hostname:3306/analyticsdb --username username --password
> password --table ml_ys_log_gmt_test --export-dir
> hdfs:///mnt/var/lib/hadoop/dfs/logs_sanitized_test
> --input-fields-terminated-by='\t'  --lines-terminated-by='\n' --verbose
>
> On my mysqlserver in the database analyticsdb, I do have the following
> table ml_ys_log_gmt_test
>
> mysql> describe ml_ys_log_gmt_test;
> +--------+-------------+------+-----+---------+-------+
> | Field  | Type        | Null | Key | Default | Extra |
> +--------+-------------+------+-----+---------+-------+
> | mydate | varchar(32) | YES  |     | NULL    |       |
> | mydata | varchar(32) | YES  |     | NULL    |       |
> +--------+-------------+------+-----+---------+-------+
>
> I can see the logs in hdfs
>
> hadoop@ip-XX-XX-XX-XX:/mnt/var/lib/hadoop/steps/5$ hadoop dfs -ls
> hdfs:///mnt/var/lib/hadoop/dfs/logs_sanitized_test
> Found 2 items
> drwxr-xr-x   - hadoop supergroup          0 2012-10-10 15:23
> /mnt/var/lib/hadoop/dfs/logs_sanitized_test/dt=2012-10-01
> drwxr-xr-x   - hadoop supergroup          0 2012-10-10 15:23
> /mnt/var/lib/hadoop/dfs/logs_sanitized_test/dt=2012-10-02
>
> and if i tail one of the file I see the correct data
>
> hadoop@ip-XX-XX-XX-XX:/mnt/var/lib/hadoop/steps/5$ hadoop dfs -tail -f
> hdfs:///mnt/var/lib/hadoop/dfs/logs_sanitized_test/dt=2012-10-01/part-m-00000
> 20121001230101 blablabla1
> 20121001230202 blablabla2
>
>
> Here is the trace when I run the command. Please note that no data get
> transferred. I would appreciate any tips. Thanks a lot!
>
> Warning: /usr/lib/hbase does not exist! HBase imports will fail.
> Please set $HBASE_HOME to the root of your HBase installation.
> 12/10/10 16:25:25 DEBUG tool.BaseSqoopTool: Enabled debug logging.
> 12/10/10 16:25:25 WARN tool.BaseSqoopTool: Setting your password on the
> command-line is insecure. Consider using -P instead.
> 12/10/10 16:25:25 DEBUG sqoop.ConnFactory: Loaded manager factory:
> com.cloudera.sqoop.manager.DefaultManagerFactory
> 12/10/10 16:25:25 DEBUG sqoop.ConnFactory: Trying ManagerFactory:
> com.cloudera.sqoop.manager.DefaultManagerFactory
> 12/10/10 16:25:25 DEBUG manager.DefaultManagerFactory: Trying with scheme:
> jdbc:mysql:
> 12/10/10 16:25:25 INFO manager.MySQLManager: Preparing to use a MySQL
> streaming resultset.
> 12/10/10 16:25:25 DEBUG sqoop.ConnFactory: Instantiated ConnManager
> org.apache.sqoop.manager.MySQLManager@5ef4f44a
> 12/10/10 16:25:25 INFO tool.CodeGenTool: Beginning code generation
> 12/10/10 16:25:25 DEBUG manager.SqlManager: No connection paramenters
> specified. Using regular API for making connection.
> 12/10/10 16:25:26 DEBUG manager.SqlManager: Using fetchSize for next query:
> -2147483648
> 12/10/10 16:25:26 INFO manager.SqlManager: Executing SQL statement: SELECT
> t.* FROM `ml_ys_log_gmt_test` AS t LIMIT 1
> 12/10/10 16:25:26 DEBUG orm.ClassWriter: selected columns:
> 12/10/10 16:25:26 DEBUG orm.ClassWriter:   mydate
> 12/10/10 16:25:26 DEBUG orm.ClassWriter:   mydata
> 12/10/10 16:25:26 DEBUG manager.SqlManager: Using fetchSize for next query:
> -2147483648
> 12/10/10 16:25:26 INFO manager.SqlManager: Executing SQL statement: SELECT
> t.* FROM `ml_ys_log_gmt_test` AS t LIMIT 1
> 12/10/10 16:25:26 DEBUG orm.ClassWriter: Writing source file:
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB