Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Sqoop, mail # dev - Review Request: Fix for SQOOP-1029


Copy link to this message
-
Re: Review Request: Fix for SQOOP-1013
Abraham Elmahrek 2013-05-31, 20:33


> On May 31, 2013, 8:30 p.m., Abraham Elmahrek wrote:
> > Seems good except for a few comments and nit picks.
> >
> > The datetime split logic is when certain intervals are "hotter" than others. IE: 1000000 rows out of 20000000 exist between the date range of december 1st to december 31st, but a user is importing the entire years data, with 12 node. Basically, 1 machine will extract 1000000 rows, while the others will extract 1000000/11 rows. In the future we could probably add some upfront analysis of the data to improve distribution.

Sorry... my wording above wasn't very concise... The datetime split logic doesn't seem to handle "hot" partitions is what I meant.
- Abraham
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/11537/#review21237
-----------------------------------------------------------
On May 30, 2013, 6:25 p.m., Venkat Ranganathan wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/11537/
> -----------------------------------------------------------
>
> (Updated May 30, 2013, 6:25 p.m.)
>
>
> Review request for Sqoop.
>
>
> Description
> -------
>
> This addresses Boolean, date, time, and timestamp splitters.
>
> THis also disallows char type splitters as discussed in SQOOP-976
>
>
> Diffs
> -----
>
>   connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcImportPartitioner.java f80f30d
>   connector/connector-generic-jdbc/src/test/java/org/apache/sqoop/connector/jdbc/TestImportPartitioner.java ee314d0
>
> Diff: https://reviews.apache.org/r/11537/diff/
>
>
> Testing
> -------
>
> Introduced new unit tests to test new functionality
> All tests pass
>
>
> Thanks,
>
> Venkat Ranganathan
>
>