Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Sqoop, mail # dev - Review Request: SQOOP-931 - Integration of Sqoop and HCatalog


Copy link to this message
-
Re: Review Request: SQOOP-931 - Integration of Sqoop and HCatalog
Venkat Ranganathan 2013-04-29, 23:21

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/10688/
-----------------------------------------------------------

(Updated April 29, 2013, 11:21 p.m.)
Review request for Sqoop and Jarek Cecho.
Changes
-------

Incorporated the review changes - create hcatalog table automatically based on command line options, hive delimiter processing and run as part of ant test target
Description
-------

This patch implements the new feature of integrating HCatalog and Sqoop.   With this feature, it is possible to import and export data between Sqoop and HCatalog tables.   The document attached to SQOOP-931 JIRA issue discusses the high level appraches.  

With this integration, more fidelity can be brought to the process of moving data between enterprise data stores and hadoop ecosystem.
Diffs (updated)
-----

  build.xml 1c33fee
  ivy.xml 1fa4dd1
  ivy/ivysettings.xml c4cc561
  src/java/org/apache/sqoop/SqoopOptions.java f18d43e
  src/java/org/apache/sqoop/config/ConfigurationConstants.java 5354063
  src/java/org/apache/sqoop/hive/HiveImport.java 838f083
  src/java/org/apache/sqoop/manager/ConnManager.java a1ac38e
  src/java/org/apache/sqoop/mapreduce/DataDrivenImportJob.java ef1d363
  src/java/org/apache/sqoop/mapreduce/ExportJobBase.java 1065d0b
  src/java/org/apache/sqoop/mapreduce/ImportJobBase.java 2465f3f
  src/java/org/apache/sqoop/mapreduce/JdbcExportJob.java 20636a0
  src/java/org/apache/sqoop/mapreduce/JobBase.java 0df1156
  src/java/org/apache/sqoop/mapreduce/hcat/SqoopHCatExportFormat.java PRE-CREATION
  src/java/org/apache/sqoop/mapreduce/hcat/SqoopHCatExportMapper.java PRE-CREATION
  src/java/org/apache/sqoop/mapreduce/hcat/SqoopHCatImportMapper.java PRE-CREATION
  src/java/org/apache/sqoop/mapreduce/hcat/SqoopHCatInputSplit.java PRE-CREATION
  src/java/org/apache/sqoop/mapreduce/hcat/SqoopHCatRecordReader.java PRE-CREATION
  src/java/org/apache/sqoop/mapreduce/hcat/SqoopHCatUtilities.java PRE-CREATION
  src/java/org/apache/sqoop/tool/BaseSqoopTool.java 9417d57
  src/java/org/apache/sqoop/tool/CodeGenTool.java dd34a97
  src/java/org/apache/sqoop/tool/ExportTool.java 215addd
  src/java/org/apache/sqoop/tool/ImportTool.java 10f0cb9
  src/perftest/ExportStressTest.java 0a41408
  src/test/com/cloudera/sqoop/hive/TestHiveImport.java 462ccf1
  src/test/com/cloudera/sqoop/testutil/BaseSqoopTestCase.java cf41b96
  src/test/com/cloudera/sqoop/testutil/ExportJobTestCase.java e13f3df
  src/test/org/apache/sqoop/hcat/HCatalogTestUtils.java PRE-CREATION
  src/test/org/apache/sqoop/hcat/TestHCatalogBasic.java PRE-CREATION
  src/test/org/apache/sqoop/hcat/TestHCatalogExport.java PRE-CREATION
  src/test/org/apache/sqoop/hcat/TestHCatalogImport.java PRE-CREATION
  testdata/hcatalog/conf/hive-log4j.properties PRE-CREATION
  testdata/hcatalog/conf/hive-site.xml PRE-CREATION
  testdata/hcatalog/conf/log4j.properties PRE-CREATION

Diff: https://reviews.apache.org/r/10688/diff/
Testing
-------

Two new integration test suites with more than 20 tests in total have been added to test various aspects of the integration.  A unit test to test the option management is also added.   All tests pass
Thanks,

Venkat Ranganathan