Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Sqoop >> mail # dev >> Review Request 12451: SQOOP-1049: Sqoop2: Record not imported if partition column value is NULL


Copy link to this message
-
Re: Review Request 12451: SQOOP-1049: Sqoop2: Record not imported if partition column value is NULL

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/12451/#review23028
-----------------------------------------------------------
Hi Mengwei,
thank you very much for identifying this issue and providing the initial patch. Would you mind adding test cases testing the new functionality?

connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcImportInitializer.java
<https://reviews.apache.org/r/12451/#comment46841>

    I'm concerned a bit of using count() aggregate function as it might lead to another full table scan which might significantly hurt performance. Maybe we could make the ability for checking nulls in the split by column optional?

connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcImportInitializer.java
<https://reviews.apache.org/r/12451/#comment46840>

    Nit: This check seems to be redundant as the query is always hardcoded and is selecting one single value.
Jarcec

- Jarek Cecho
On July 10, 2013, 7:02 p.m., Mengwei Ding wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/12451/
> -----------------------------------------------------------
>
> (Updated July 10, 2013, 7:02 p.m.)
>
>
> Review request for Sqoop and Jarek Cecho.
>
>
> Bugs: SQOOP-1049
>     https://issues.apache.org/jira/browse/SQOOP-1049
>
>
> Repository: sqoop-sqoop2
>
>
> Description
> -------
>
> commit 47e73c30b49be0168459d76bf8993205c7a4f4fc
> Author: Mengwei Ding <[EMAIL PROTECTED]>
> Date:   Wed Jul 10 11:41:05 2013 -0700
>
>     SQOOP-1049: Sqoop2: Record not imported if partition column value is NULL
>
> :100644 100644 abcc89d... a940d15... M connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcConnectorConstants.java
> :100644 100644 671bb4a... d331ae8... M connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcConnectorError.java
> :100644 100644 96818ba... 357fefb... M connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcImportInitializer.java
> :100644 100644 4401800... ff80ed3... M connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcImportPartitioner.java
>
>
> Diffs
> -----
>
>   connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcConnectorConstants.java abcc89d
>   connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcConnectorError.java 671bb4a
>   connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcImportInitializer.java 96818ba
>   connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcImportPartitioner.java 4401800
>
> Diff: https://reviews.apache.org/r/12451/diff/
>
>
> Testing
> -------
>
> Have done a manual test, in which I successfully import a table with some null values in partition column.
>
>
> Thanks,
>
> Mengwei Ding
>
>