Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Sqoop >> mail # dev >> Review Request: SQOOP-931 - Integration of Sqoop and HCatalog


+
Venkat Ranganathan 2013-04-21, 05:51
+
Venkat Ranganathan 2013-04-24, 05:13
+
Venkat Ranganathan 2013-04-29, 23:21
+
Venkat Ranganathan 2013-04-30, 06:56
+
Venkat Ranganathan 2013-05-04, 23:46
+
Jarek Cecho 2013-05-20, 13:02
+
Venkat Ranganathan 2013-05-21, 00:35
+
Jarek Cecho 2013-05-21, 10:09
+
Venkat Ranganathan 2013-05-24, 23:18
+
Jarek Cecho 2013-05-28, 09:33
+
Venkat Ranganathan 2013-05-28, 20:38
+
Venkat Ranganathan 2013-05-29, 20:55
+
Venkat Ranganathan 2013-06-02, 20:33
+
Venkat Ranganathan 2013-06-03, 04:16
+
Jarek Cecho 2013-06-04, 23:15
+
Venkat Ranganathan 2013-06-05, 00:09
+
Venkat Ranganathan 2013-06-05, 03:52
+
Venkat Ranganathan 2013-06-05, 21:42
+
Jarek Cecho 2013-06-05, 21:26
Copy link to this message
-
Re: Review Request: SQOOP-931 - Integration of Sqoop and HCatalog

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/10688/
-----------------------------------------------------------

(Updated June 6, 2013, midnight)
Review request for Sqoop and Jarek Cecho.
Changes
-------

New review changes.    Fixed documentation and added a new test to validate we fail when create-hcatalog-table is provided with preexisting table.

Removed inline hcat client execution.   It causes issues with logger configuration being reset by hive and specifying hiveconfiguration on the command line will entail more significant changes.

For tests we still use inline hcat execution.
Description
-------

This patch implements the new feature of integrating HCatalog and Sqoop.   With this feature, it is possible to import and export data between Sqoop and HCatalog tables.   The document attached to SQOOP-931 JIRA issue discusses the high level appraches.  

With this integration, more fidelity can be brought to the process of moving data between enterprise data stores and hadoop ecosystem.
Diffs (updated)
-----

  bin/configure-sqoop 61ff3f2
  build.xml 636c103
  ivy.xml 1fa4dd1
  ivy/ivysettings.xml c4cc561
  src/docs/user/SqoopUserGuide.txt 01ac1cf
  src/docs/user/hcatalog.txt PRE-CREATION
  src/java/org/apache/sqoop/SqoopOptions.java f18d43e
  src/java/org/apache/sqoop/config/ConfigurationConstants.java 5354063
  src/java/org/apache/sqoop/hive/HiveImport.java 838f083
  src/java/org/apache/sqoop/manager/ConnManager.java a1ac38e
  src/java/org/apache/sqoop/mapreduce/DataDrivenImportJob.java ef1d363
  src/java/org/apache/sqoop/mapreduce/ExportJobBase.java 1065d0b
  src/java/org/apache/sqoop/mapreduce/ImportJobBase.java 2465f3f
  src/java/org/apache/sqoop/mapreduce/JdbcExportJob.java 20636a0
  src/java/org/apache/sqoop/mapreduce/JobBase.java 0df1156
  src/java/org/apache/sqoop/mapreduce/hcat/SqoopHCatExportFormat.java PRE-CREATION
  src/java/org/apache/sqoop/mapreduce/hcat/SqoopHCatExportMapper.java PRE-CREATION
  src/java/org/apache/sqoop/mapreduce/hcat/SqoopHCatImportMapper.java PRE-CREATION
  src/java/org/apache/sqoop/mapreduce/hcat/SqoopHCatInputSplit.java PRE-CREATION
  src/java/org/apache/sqoop/mapreduce/hcat/SqoopHCatRecordReader.java PRE-CREATION
  src/java/org/apache/sqoop/mapreduce/hcat/SqoopHCatUtilities.java PRE-CREATION
  src/java/org/apache/sqoop/tool/BaseSqoopTool.java 42f521f
  src/java/org/apache/sqoop/tool/CodeGenTool.java dd34a97
  src/java/org/apache/sqoop/tool/ExportTool.java 215addd
  src/java/org/apache/sqoop/tool/ImportTool.java 2627726
  src/perftest/ExportStressTest.java 0a41408
  src/test/com/cloudera/sqoop/ThirdPartyTests.java 06f7122
  src/test/com/cloudera/sqoop/hive/TestHiveImport.java 462ccf1
  src/test/com/cloudera/sqoop/testutil/BaseSqoopTestCase.java cf41b96
  src/test/com/cloudera/sqoop/testutil/ExportJobTestCase.java e13f3df
  src/test/org/apache/sqoop/hcat/HCatalogExportTest.java PRE-CREATION
  src/test/org/apache/sqoop/hcat/HCatalogImportTest.java PRE-CREATION
  src/test/org/apache/sqoop/hcat/HCatalogTestUtils.java PRE-CREATION
  src/test/org/apache/sqoop/hcat/TestHCatalogBasic.java PRE-CREATION
  testdata/hcatalog/conf/hive-log4j.properties PRE-CREATION
  testdata/hcatalog/conf/hive-site.xml PRE-CREATION
  testdata/hcatalog/conf/log4j.properties PRE-CREATION

Diff: https://reviews.apache.org/r/10688/diff/
Testing
-------

Two new integration test suites with more than 20 tests in total have been added to test various aspects of the integration.  A unit test to test the option management is also added.   All tests pass
Thanks,

Venkat Ranganathan

+
Venkat Ranganathan 2013-06-06, 22:55
+
Venkat Ranganathan 2013-06-07, 02:03
+
Jarek Cecho 2013-06-07, 14:29
+
Jarek Cecho 2013-06-07, 01:03
+
Venkat Ranganathan 2013-06-07, 01:53
+
Jarek Cecho 2013-06-06, 18:34
+
Venkat Ranganathan 2013-06-06, 19:07