Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Sqoop >> mail # user >> Sqoop Import from postgres with option hive-overwrite is failing


Copy link to this message
-
Re: Sqoop Import from postgres with option hive-overwrite is failing
Hi Gopi
     You are encountering this issue because the target-dir/output dir for
sqoop import already exists. Sqoop first imports data to a hdfs directory
(specified by target dir) and then LOADS the same to hive warehouse
directory using LOAD DATA statement. Specifying a different target-dir
other than the hive warehouse directory should resolve your issue.

Regards
Bejoy KS

On Thu, Apr 26, 2012 at 11:32 AM, Gopi Kodumur <[EMAIL PROTECTED]> wrote:

> I'm trying to import a postgres table to hive using --hive-overwrite is
> failing..Any help ?
>
>
> Sqoop  Version 1.3.0-cdh3u2
> git commit id f3f0f8efda47f2cfe7464d1493b6af48ecd44a15
> &
>
> hive 0.6
>
>  sqoop import --options-file /tmp/hive_import --table postgres_table1
> --hive-import --hive-table hive_Table1 --hive-overwrite --target-dir
> /user/hive/warehouse/hive_Table1
>  --direct --where "event_dt='2012-04-14'"   --verbose -m 1
>
> Warning: /usr/lib/hbase does not exist! HBase imports will fail.
> Please set $HBASE_HOME to the root of your HBase installation.
> 12/04/26 05:25:15 DEBUG tool.BaseSqoopTool: Enabled debug logging.
> 12/04/26 05:25:15 INFO tool.BaseSqoopTool: Using Hive-specific delimiters
> for output. You can override
> 12/04/26 05:25:15 INFO tool.BaseSqoopTool: delimiters with
> --fields-terminated-by, etc.
> 12/04/26 05:25:15 DEBUG sqoop.ConnFactory: Loaded manager factory:
> com.cloudera.sqoop.manager.DefaultManagerFactory
> 12/04/26 05:25:15 DEBUG sqoop.ConnFactory: Trying ManagerFactory:
> com.cloudera.sqoop.manager.DefaultManagerFactory
> 12/04/26 05:25:15 DEBUG manager.DefaultManagerFactory: Trying with scheme:
> jdbc:postgresql:
> 12/04/26 05:25:15 INFO manager.SqlManager: Using default fetchSize of 1000
> 12/04/26 05:25:15 DEBUG sqoop.ConnFactory: Instantiated ConnManager
> com.cloudera.sqoop.manager.DirectPostgresqlManager@1be1a408
> 12/04/26 05:25:15 INFO tool.CodeGenTool: Beginning code generation
> 12/04/26 05:25:15 DEBUG manager.SqlManager: No connection paramenters
> specified. Using regular API for making connection.
> 12/04/26 05:25:15 DEBUG manager.SqlManager: Using fetchSize for next
> query: 1000
> 12/04/26 05:25:15 INFO manager.SqlManager: Executing SQL statement: SELECT
> t.* FROM "postgres_table1" AS t LIMIT 1
> 12/04/26 05:25:15 DEBUG orm.ClassWriter: selected columns:
> 12/04/26 05:25:15 DEBUG orm.ClassWriter:   event_date
> 12/04/26 05:25:15 DEBUG orm.ClassWriter:   event_dt
> 12/04/26 05:25:15 DEBUG orm.ClassWriter:   event_id
> 12/04/26 05:25:15 DEBUG orm.ClassWriter:   user_id
> 12/04/26 05:25:15 DEBUG orm.ClassWriter:   ip
> 12/04/26 05:25:15 DEBUG orm.ClassWriter:   advertiser_id
> 12/04/26 05:25:15 DEBUG orm.ClassWriter:   order_id
> 12/04/26 05:25:15 DEBUG orm.ClassWriter:   ad_id
> 12/04/26 05:25:15 DEBUG orm.ClassWriter:   creative_id
> 12/04/26 05:25:15 DEBUG orm.ClassWriter:   creative_version
> 12/04/26 05:25:15 DEBUG orm.ClassWriter:   creative_size_id
> 12/04/26 05:25:15 DEBUG orm.ClassWriter:   site_id
> 12/04/26 05:25:15 DEBUG orm.ClassWriter:   page_id
> 12/04/26 05:25:15 DEBUG orm.ClassWriter:   keyword
> 12/04/26 05:25:15 DEBUG orm.ClassWriter:   country_id
> 12/04/26 05:25:15 DEBUG orm.ClassWriter:   state_province
> 12/04/26 05:25:15 DEBUG orm.ClassWriter:   browser_id
> 12/04/26 05:25:15 DEBUG orm.ClassWriter:   browser_version
> 12/04/26 05:25:15 DEBUG orm.ClassWriter:   os_id
> 12/04/26 05:25:15 DEBUG orm.ClassWriter:   dma_id
> 12/04/26 05:25:15 DEBUG orm.ClassWriter:   city_id
> 12/04/26 05:25:15 DEBUG orm.ClassWriter:   site_data
> 12/04/26 05:25:15 DEBUG orm.ClassWriter:   cre_date
> 12/04/26 05:25:15 DEBUG orm.ClassWriter:   cre_user
> 12/04/26 05:25:15 DEBUG orm.ClassWriter: Writing source file:
> /tmp/sqoop-etl_user/compile/3899eebe45d0d9bafc712d0e4795b58b/postgres_table1.java
> 12/04/26 05:25:15 DEBUG orm.ClassWriter: Table name: postgres_table1
> 12/04/26 05:25:15 DEBUG orm.ClassWriter: Columns: event_date:93,
> event_dt:91, event_id:12, user_id:2, ip:12, advertiser_id:4, order_id:4,
> ad_id:4, creative_id:2, creative_version:4, creative_size_id:12, site_id:4,
Regards
       Bejoy