Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Sqoop >> mail # user >> sqoop-1.4.2 --export-dir


Copy link to this message
-
Re: sqoop-1.4.2 --export-dir
Hi Matthieu,
You're right, parameter --export-dir is not doing any recursive lookup for files. It expects directory with files that you want export.

Jarcec

On Fri, Mar 15, 2013 at 01:36:50PM -0400, Matthieu Labour wrote:
> Hi
>
> I am using sqoop-1.4.2 to export to Sql. Thank you for creating this too.
>
> My question concerns the use of --export-dir. It seems that --export-dir
> does not work when the dir being passed is the root of subdirectories.
> Please can you advise? Thank you.
>
> Example:
>
> The following command exports the data to Sql
>
> ~/sqoop-1.4.2.bin__hadoop-1.0.0/bin/sqoop export --connect
> jdbc:postgresql://ec2-XX-XXX-XXX-XXX.compute-1.amazonaws.com:XXXX/xxxxxxxxxxxxx
> --username xxxxxxxxxx --password xxxxxxxxxx --table
> ml_ys_log_gmt_daily_experiment_2 --export-dir
> =hdfs:///mnt/var/lib/hadoop/dfs/logs_daily_sanitized/dt=2013-02-01
> --input-fields-terminated-by='\t' --lines-terminated-by='\n' --verbose
> --batch
>
>
> hadoop@domU-XX-XX-XX-XX-XX-XX:/mnt/var/lib/hadoop/steps/2$ hadoop fs -ls
> hdfs:///mnt/var/lib/hadoop/dfs/logs_daily_sanitized/dt=2013-02-01
> Found 1 items
> -rw-r--r--   1 hadoop supergroup   15931406 2013-03-15 17:03
> /mnt/var/lib/hadoop/dfs/logs_daily_sanitized/dt=2013-02-01/part-r-00001
>
> The following command does not export the data to Sql
>
> ~/sqoop-1.4.2.bin__hadoop-1.0.0/bin/sqoop export --connect
> jdbc:postgresql://ec2-XX-XXX-XXX-XXX.compute-1.amazonaws.com:XXXX/xxxxxxxxxxxxx
> --username xxxxxxxxxx --password xxxxxxxxxx --table
> ml_ys_log_gmt_daily_experiment_2 --export-dir
> =hdfs:///mnt/var/lib/hadoop/dfs/logs_daily_sanitized
> --input-fields-terminated-by='\t' --lines-terminated-by='\n' --verbose
> --batch
>
> hadoop@domU-XX-XX-XX-XX-XX-XX:/mnt/var/lib/hadoop/steps/2$ hadoop fs -ls
> hdfs:///mnt/var/lib/hadoop/dfs/logs_daily_sanitized/
> Found 44 items
> -rw-r--r--   1 hadoop supergroup          0 2013-03-15 17:03
> /mnt/var/lib/hadoop/dfs/logs_daily_sanitized/_SUCCESS
> drwxr-xr-x   - hadoop supergroup          0 2013-03-15 17:03
> /mnt/var/lib/hadoop/dfs/logs_daily_sanitized/dt=2013-02-01
> drwxr-xr-x   - hadoop supergroup          0 2013-03-15 17:03
> /mnt/var/lib/hadoop/dfs/logs_daily_sanitized/dt=2013-02-02
> drwxr-xr-x   - hadoop supergroup          0 2013-03-15 17:03
> /mnt/var/lib/hadoop/dfs/logs_daily_sanitized/dt=2013-02-03
> drwxr-xr-x   - hadoop supergroup          0 2013-03-15 17:03
> /mnt/var/lib/hadoop/dfs/logs_daily_sanitized/dt=2013-02-04
> drwxr-xr-x   - hadoop supergroup          0 2013-03-15 17:03
> /mnt/var/lib/hadoop/dfs/logs_daily_sanitized/dt=2013-02-05
> drwxr-xr-x   - hadoop supergroup          0 2013-03-15 17:03
> /mnt/var/lib/hadoop/dfs/logs_daily_sanitized/dt=2013-02-06
> drwxr-xr-x   - hadoop supergroup          0 2013-03-15 17:03
> /mnt/var/lib/hadoop/dfs/logs_daily_sanitized/dt=2013-02-07
> drwxr-xr-x   - hadoop supergroup          0 2013-03-15 17:03
> /mnt/var/lib/hadoop/dfs/logs_daily_sanitized/dt=2013-02-08
> drwxr-xr-x   - hadoop supergroup          0 2013-03-15 17:03
> /mnt/var/lib/hadoop/dfs/logs_daily_sanitized/dt=2013-02-09
> drwxr-xr-x   - hadoop supergroup          0 2013-03-15 17:03
> /mnt/var/lib/hadoop/dfs/logs_daily_sanitized/dt=2013-02-10
> drwxr-xr-x   - hadoop supergroup          0 2013-03-15 17:03
> /mnt/var/lib/hadoop/dfs/logs_daily_sanitized/dt=2013-02-11
> drwxr-xr-x   - hadoop supergroup          0 2013-03-15 17:03
> /mnt/var/lib/hadoop/dfs/logs_daily_sanitized/dt=2013-02-12
> drwxr-xr-x   - hadoop supergroup          0 2013-03-15 17:04
> /mnt/var/lib/hadoop/dfs/logs_daily_sanitized/dt=2013-02-13
> drwxr-xr-x   - hadoop supergroup          0 2013-03-15 17:04
> /mnt/var/lib/hadoop/dfs/logs_daily_sanitized/dt=2013-02-14
> drwxr-xr-x   - hadoop supergroup          0 2013-03-15 17:04
> /mnt/var/lib/hadoop/dfs/logs_daily_sanitized/dt=2013-02-15
> drwxr-xr-x   - hadoop supergroup          0 2013-03-15 17:04
> /mnt/var/lib/hadoop/dfs/logs_daily_sanitized/dt=2013-02-16
> drwxr-xr-x   - hadoop supergroup          0 2013-03-15 17:04
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB