Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Sqoop, mail # dev - Review Request: SQOOP-390: PostgreSQL connector for direct export with pg_bulkload


Copy link to this message
-
Re: Review Request: SQOOP-390: PostgreSQL connector for direct export with pg_bulkload
Masatake Iwasaki 2012-07-30, 00:57


> On July 26, 2012, 9:13 p.m., Jarek Cecho wrote:
> > /src/java/org/apache/sqoop/mapreduce/PGBulkloadExportReducer.java, line 83
> > <https://reviews.apache.org/r/2724/diff/3/?file=129307#file129307line83>
> >
> >     I'm thinking whether it would be beneficial here not to die immediately. For example if user is exporting 30 partitions and there will be some issues with the first. He would be forced to manually do all 30 of them. But if we would try all them first and log only the failed ones, user would have to fix only that one. What do you think sir?

The motivation for the current design is keeping destination table clean by staging in a single atomic transaction. I think making a number of reduce tasks configurable is a good solution. Users can choose single atomic staging or parallel staging.
- Masatake
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2724/#review9514
-----------------------------------------------------------
On July 26, 2012, 10:41 a.m., Masatake Iwasaki wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/2724/
> -----------------------------------------------------------
>
> (Updated July 26, 2012, 10:41 a.m.)
>
>
> Review request for Sqoop.
>
>
> Description
> -------
>
> Patch for SQOOP-390
> https://issues.apache.org/jira/browse/SQOOP-390
>
>
> This addresses bug SQOOP-390.
>     https://issues.apache.org/jira/browse/SQOOP-390
>
>
> Diffs
> -----
>
>   /src/java/org/apache/sqoop/manager/PGBulkloadManager.java PRE-CREATION
>   /src/java/org/apache/sqoop/mapreduce/AutoProgressReducer.java PRE-CREATION
>   /src/java/org/apache/sqoop/mapreduce/PGBulkloadExportJob.java PRE-CREATION
>   /src/java/org/apache/sqoop/mapreduce/PGBulkloadExportMapper.java PRE-CREATION
>   /src/java/org/apache/sqoop/mapreduce/PGBulkloadExportReducer.java PRE-CREATION
>   /src/test/com/cloudera/sqoop/manager/PGBulkloadManagerManualTest.java PRE-CREATION
>
> Diff: https://reviews.apache.org/r/2724/diff/
>
>
> Testing
> -------
>
> This patch include the test class PGBulkloadManagerTest.
> I've tested "ant test" and passed.
>
>
> Thanks,
>
> Masatake Iwasaki
>
>