Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Sqoop >> mail # dev >> Review Request 12936: SQOOP-777. Sqoop2: Pluggable Intermediate Data Format


+
Hari Shreedharan 2013-07-25, 04:18
Copy link to this message
-
Re: Review Request 12936: SQOOP-777. Sqoop2: Pluggable Intermediate Data Format

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/12936/
-----------------------------------------------------------

(Updated July 26, 2013, 12:04 a.m.)
Review request for Sqoop.
Changes
-------

Unit tests for SqoopWritable. Some code cleanup
Bugs: SQOOP-777
    https://issues.apache.org/jira/browse/SQOOP-777
Repository: sqoop-sqoop2
Description
-------

Implemented a pluggable intermediate data format that decouples the internal representation of the data from the connector and the output formats. Connectors can choose to implement and support a format that is more efficient for them. Also separated the SqoopWritable so that we can use the intermediate data format independent of (current) Hadoop.

I ran a full build - all tests including integration tests pass. I have not added any new tests, yet. I will add unit tests for the new classes. Also, I have not tried running this on an actual cluster - so things may be broken. I'd like some initial feedback based on the current patch.

I also implemented escaping of characters. There is some work remaining to support binary format, but it is mostly integration, the basic implementation is in place.
Diffs (updated)
-----

  common/pom.xml db11b5b
  common/src/main/java/org/apache/sqoop/etl/io/CSVIntermediateDataFormat.java PRE-CREATION
  common/src/main/java/org/apache/sqoop/etl/io/DataReader.java 3e1adc7
  common/src/main/java/org/apache/sqoop/etl/io/DataWriter.java d81364e
  common/src/main/java/org/apache/sqoop/etl/io/IntermediateDataFormat.java PRE-CREATION
  common/src/main/java/org/apache/sqoop/schema/type/Column.java 8b630b2
  connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcConnector.java e0da80f
  connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcExportInitializer.java 7212843
  connector/connector-generic-jdbc/src/test/java/org/apache/sqoop/connector/jdbc/TestExportLoader.java aa1c4ff
  connector/connector-generic-jdbc/src/test/java/org/apache/sqoop/connector/jdbc/TestImportExtractor.java a7ed6ba
  core/src/main/java/org/apache/sqoop/framework/FrameworkConstants.java 4293dce
  core/src/main/java/org/apache/sqoop/framework/JobManager.java 9f09982
  core/src/main/java/org/apache/sqoop/framework/SubmissionRequest.java 53d0039
  execution/mapreduce/pom.xml f9a2a0e
  execution/mapreduce/src/main/java/org/apache/sqoop/execution/mapreduce/MapreduceExecutionEngine.java 767080c
  execution/mapreduce/src/main/java/org/apache/sqoop/job/etl/HdfsExportExtractor.java 1978ec6
  execution/mapreduce/src/main/java/org/apache/sqoop/job/etl/HdfsSequenceImportLoader.java a07c511
  execution/mapreduce/src/main/java/org/apache/sqoop/job/etl/HdfsTextImportLoader.java 4621942
  execution/mapreduce/src/main/java/org/apache/sqoop/job/io/SqoopWritable.java PRE-CREATION
  execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopFileOutputFormat.java 356ae8a
  execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopMapper.java 59cf391
  execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopNullOutputFormat.java 90de6ef
  execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopOutputFormatLoadExecutor.java 739eb17
  execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopReducer.java b31161c
  execution/mapreduce/src/test/java/org/apache/sqoop/job/JobUtils.java e21f15b
  execution/mapreduce/src/test/java/org/apache/sqoop/job/TestHdfsExtract.java b7079dd
  execution/mapreduce/src/test/java/org/apache/sqoop/job/TestHdfsLoad.java f849aae
  execution/mapreduce/src/test/java/org/apache/sqoop/job/TestMapReduce.java 7b264c6
  execution/mapreduce/src/test/java/org/apache/sqoop/job/io/SqoopWritableTest.java PRE-CREATION
  execution/mapreduce/src/test/java/org/apache/sqoop/job/mr/TestSqoopOutputFormatLoadExecutor.java bee8ab7
  pom.xml 520c107
  spi/src/main/java/org/apache/sqoop/connector/spi/SqoopConnector.java 2becc56
  submission/mapreduce/src/main/java/org/apache/sqoop/submission/mapreduce/MapreduceSubmissionEngine.java 6fc485b

Diff: https://reviews.apache.org/r/12936/diff/
Testing
Thanks,

Hari Shreedharan
+
Hari Shreedharan 2013-07-27, 01:46
+
Hari Shreedharan 2013-07-27, 04:11
+
Hari Shreedharan 2013-08-01, 03:41
+
Venkat Ranganathan 2013-08-01, 04:42
+
Hari Shreedharan 2013-08-01, 18:34
+
Venkat Ranganathan 2013-08-01, 18:56
+
Hari Shreedharan 2013-08-01, 20:24
+
Jarek Cecho 2013-08-26, 23:55
+
Jarek Cecho 2013-07-31, 23:25
+
Hari Shreedharan 2013-08-01, 03:40