Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Sqoop >> mail # dev >> Review Request: SQOOP-931 - Integration of Sqoop and HCatalog


Copy link to this message
-
Re: Review Request: SQOOP-931 - Integration of Sqoop and HCatalog

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/10688/
-----------------------------------------------------------

(Updated April 24, 2013, 5:12 a.m.)
Review request for Sqoop and Jarek Cecho.
Changes
-------

Fixed the mappers' key type to be WritableComparable instead of a specific keytype so that we can handle tables with different storage formats.  Also added new tests for sequence and text format files
Description
-------

This patch implements the new feature of integrating HCatalog and Sqoop.   With this feature, it is possible to import and export data between Sqoop and HCatalog tables.   The document attached to SQOOP-931 JIRA issue discusses the high level appraches.  

With this integration, more fidelity can be brought to the process of moving data between enterprise data stores and hadoop ecosystem.
Diffs (updated)
-----

  build.xml 1c33fee
  ivy.xml 1fa4dd1
  src/java/org/apache/sqoop/SqoopOptions.java f18d43e
  src/java/org/apache/sqoop/config/ConfigurationConstants.java 5354063
  src/java/org/apache/sqoop/manager/ConnManager.java a1ac38e
  src/java/org/apache/sqoop/mapreduce/DataDrivenImportJob.java ef1d363
  src/java/org/apache/sqoop/mapreduce/ExportJobBase.java 1065d0b
  src/java/org/apache/sqoop/mapreduce/ImportJobBase.java 2465f3f
  src/java/org/apache/sqoop/mapreduce/JdbcExportJob.java 20636a0
  src/java/org/apache/sqoop/mapreduce/hcat/SqoopHCatExportFormat.java PRE-CREATION
  src/java/org/apache/sqoop/mapreduce/hcat/SqoopHCatExportMapper.java PRE-CREATION
  src/java/org/apache/sqoop/mapreduce/hcat/SqoopHCatImportMapper.java PRE-CREATION
  src/java/org/apache/sqoop/mapreduce/hcat/SqoopHCatInputSplit.java PRE-CREATION
  src/java/org/apache/sqoop/mapreduce/hcat/SqoopHCatRecordReader.java PRE-CREATION
  src/java/org/apache/sqoop/mapreduce/hcat/SqoopHCatUtilities.java PRE-CREATION
  src/java/org/apache/sqoop/tool/BaseSqoopTool.java 9417d57
  src/java/org/apache/sqoop/tool/CodeGenTool.java dd34a97
  src/java/org/apache/sqoop/tool/ExportTool.java 215addd
  src/java/org/apache/sqoop/tool/ImportTool.java 10f0cb9
  src/perftest/ExportStressTest.java 0a41408
  src/test/com/cloudera/sqoop/TestHCatalogBasic.java PRE-CREATION
  src/test/com/cloudera/sqoop/hcat/HCatalogExportManualTest.java PRE-CREATION
  src/test/com/cloudera/sqoop/hcat/HCatalogImportManualTest.java PRE-CREATION
  src/test/com/cloudera/sqoop/hcat/HCatalogTestUtils.java PRE-CREATION

Diff: https://reviews.apache.org/r/10688/diff/
Testing
-------

Two new integration test suites with more than 20 tests in total have been added to test various aspects of the integration.  A unit test to test the option management is also added.   All tests pass
Thanks,

Venkat Ranganathan