Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Sqoop >> mail # user >> Sqoop Import from postgres with option hive-overwrite is failing


Copy link to this message
-
Re: Sqoop Import from postgres with option hive-overwrite is failing
Hi Gopi
     You are encountering this issue because the target-dir/output dir for
sqoop import already exists. Sqoop first imports data to a hdfs directory
(specified by target dir) and then LOADS the same to hive warehouse
directory using LOAD DATA statement. Specifying a different target-dir
other than the hive warehouse directory should resolve your issue.

Regards
Bejoy KS

On Thu, Apr 26, 2012 at 11:32 AM, Gopi Kodumur <[EMAIL PROTECTED]> wrote:

> I'm trying to import a postgres table to hive using --hive-overwrite is
> failing..Any help ?
>
>
> Sqoop  Version 1.3.0-cdh3u2
> git commit id f3f0f8efda47f2cfe7464d1493b6af48ecd44a15
> &
>
> hive 0.6
>
>  sqoop import --options-file /tmp/hive_import --table postgres_table1
> --hive-import --hive-table hive_Table1 --hive-overwrite --target-dir
> /user/hive/warehouse/hive_Table1
>  --direct --where "event_dt='2012-04-14'"   --verbose -m 1
>
> Warning: /usr/lib/hbase does not exist! HBase imports will fail.
> Please set $HBASE_HOME to the root of your HBase installation.
> 12/04/26 05:25:15 DEBUG tool.BaseSqoopTool: Enabled debug logging.
> 12/04/26 05:25:15 INFO tool.BaseSqoopTool: Using Hive-specific delimiters
> for output. You can override
> 12/04/26 05:25:15 INFO tool.BaseSqoopTool: delimiters with
> --fields-terminated-by, etc.
> 12/04/26 05:25:15 DEBUG sqoop.ConnFactory: Loaded manager factory:
> com.cloudera.sqoop.manager.DefaultManagerFactory
> 12/04/26 05:25:15 DEBUG sqoop.ConnFactory: Trying ManagerFactory:
> com.cloudera.sqoop.manager.DefaultManagerFactory
> 12/04/26 05:25:15 DEBUG manager.DefaultManagerFactory: Trying with scheme:
> jdbc:postgresql:
> 12/04/26 05:25:15 INFO manager.SqlManager: Using default fetchSize of 1000
> 12/04/26 05:25:15 DEBUG sqoop.ConnFactory: Instantiated ConnManager
> com.cloudera.sqoop.manager.DirectPostgresqlManager@1be1a408
> 12/04/26 05:25:15 INFO tool.CodeGenTool: Beginning code generation
> 12/04/26 05:25:15 DEBUG manager.SqlManager: No connection paramenters
> specified. Using regular API for making connection.
> 12/04/26 05:25:15 DEBUG manager.SqlManager: Using fetchSize for next
> query: 1000
> 12/04/26 05:25:15 INFO manager.SqlManager: Executing SQL statement: SELECT
> t.* FROM "postgres_table1" AS t LIMIT 1
> 12/04/26 05:25:15 DEBUG orm.ClassWriter: selected columns:
> 12/04/26 05:25:15 DEBUG orm.ClassWriter:   event_date
> 12/04/26 05:25:15 DEBUG orm.ClassWriter:   event_dt
> 12/04/26 05:25:15 DEBUG orm.ClassWriter:   event_id
> 12/04/26 05:25:15 DEBUG orm.ClassWriter:   user_id
> 12/04/26 05:25:15 DEBUG orm.ClassWriter:   ip
> 12/04/26 05:25:15 DEBUG orm.ClassWriter:   advertiser_id
> 12/04/26 05:25:15 DEBUG orm.ClassWriter:   order_id
> 12/04/26 05:25:15 DEBUG orm.ClassWriter:   ad_id
> 12/04/26 05:25:15 DEBUG orm.ClassWriter:   creative_id
> 12/04/26 05:25:15 DEBUG orm.ClassWriter:   creative_version
> 12/04/26 05:25:15 DEBUG orm.ClassWriter:   creative_size_id
> 12/04/26 05:25:15 DEBUG orm.ClassWriter:   site_id
> 12/04/26 05:25:15 DEBUG orm.ClassWriter:   page_id
> 12/04/26 05:25:15 DEBUG orm.ClassWriter:   keyword
> 12/04/26 05:25:15 DEBUG orm.ClassWriter:   country_id
> 12/04/26 05:25:15 DEBUG orm.ClassWriter:   state_province
> 12/04/26 05:25:15 DEBUG orm.ClassWriter:   browser_id
> 12/04/26 05:25:15 DEBUG orm.ClassWriter:   browser_version
> 12/04/26 05:25:15 DEBUG orm.ClassWriter:   os_id
> 12/04/26 05:25:15 DEBUG orm.ClassWriter:   dma_id
> 12/04/26 05:25:15 DEBUG orm.ClassWriter:   city_id
> 12/04/26 05:25:15 DEBUG orm.ClassWriter:   site_data
> 12/04/26 05:25:15 DEBUG orm.ClassWriter:   cre_date
> 12/04/26 05:25:15 DEBUG orm.ClassWriter:   cre_user
> 12/04/26 05:25:15 DEBUG orm.ClassWriter: Writing source file:
> /tmp/sqoop-etl_user/compile/3899eebe45d0d9bafc712d0e4795b58b/postgres_table1.java
> 12/04/26 05:25:15 DEBUG orm.ClassWriter: Table name: postgres_table1
> 12/04/26 05:25:15 DEBUG orm.ClassWriter: Columns: event_date:93,
> event_dt:91, event_id:12, user_id:2, ip:12, advertiser_id:4, order_id:4,
> ad_id:4, creative_id:2, creative_version:4, creative_size_id:12, site_id:4,
Regards
       Bejoy
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB