Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Sqoop, mail # dev - Review Request: SQOOP-390: PostgreSQL connector for direct export with pg_bulkload


Copy link to this message
-
Re: Review Request: SQOOP-390: PostgreSQL connector for direct export with pg_bulkload
Masatake Iwasaki 2012-07-30, 03:25


> On July 27, 2012, 4:34 p.m., Jarek Cecho wrote:
> > /src/java/org/apache/sqoop/mapreduce/PGBulkloadExportMapper.java, lines 82-86
> > <https://reviews.apache.org/r/2724/diff/3/?file=129306#file129306line82>
> >
> >     Could add option to create those temporary tables in different database?

As far as PostgreSQL concerned, staging across databases is inefficient because it causes network data transfer via client (slave node). Also this change requires handling of multiple connections and causes a lot of code modifications.  I would like to leave this as a future improvement.
It may be more preferable to handle the feature connecting to multiple databases for staging in a independent JIRA issue about Sqoop global functionality.
- Masatake
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2724/#review9540
-----------------------------------------------------------
On July 26, 2012, 10:41 a.m., Masatake Iwasaki wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/2724/
> -----------------------------------------------------------
>
> (Updated July 26, 2012, 10:41 a.m.)
>
>
> Review request for Sqoop.
>
>
> Description
> -------
>
> Patch for SQOOP-390
> https://issues.apache.org/jira/browse/SQOOP-390
>
>
> This addresses bug SQOOP-390.
>     https://issues.apache.org/jira/browse/SQOOP-390
>
>
> Diffs
> -----
>
>   /src/java/org/apache/sqoop/manager/PGBulkloadManager.java PRE-CREATION
>   /src/java/org/apache/sqoop/mapreduce/AutoProgressReducer.java PRE-CREATION
>   /src/java/org/apache/sqoop/mapreduce/PGBulkloadExportJob.java PRE-CREATION
>   /src/java/org/apache/sqoop/mapreduce/PGBulkloadExportMapper.java PRE-CREATION
>   /src/java/org/apache/sqoop/mapreduce/PGBulkloadExportReducer.java PRE-CREATION
>   /src/test/com/cloudera/sqoop/manager/PGBulkloadManagerManualTest.java PRE-CREATION
>
> Diff: https://reviews.apache.org/r/2724/diff/
>
>
> Testing
> -------
>
> This patch include the test class PGBulkloadManagerTest.
> I've tested "ant test" and passed.
>
>
> Thanks,
>
> Masatake Iwasaki
>
>