Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Sqoop >> mail # dev >> Review Request: SQOOP-931 - Integration of Sqoop and HCatalog


+
Venkat Ranganathan 2013-04-21, 05:51
+
Venkat Ranganathan 2013-04-24, 05:13
+
Venkat Ranganathan 2013-04-29, 23:21
+
Venkat Ranganathan 2013-04-30, 06:56
+
Venkat Ranganathan 2013-05-04, 23:46
+
Jarek Cecho 2013-05-20, 13:02
+
Venkat Ranganathan 2013-05-21, 00:35
+
Jarek Cecho 2013-05-21, 10:09
+
Venkat Ranganathan 2013-05-24, 23:18
+
Jarek Cecho 2013-05-28, 09:33
+
Venkat Ranganathan 2013-05-28, 20:38
+
Venkat Ranganathan 2013-05-29, 20:55
+
Venkat Ranganathan 2013-06-02, 20:33
+
Venkat Ranganathan 2013-06-03, 04:16
+
Jarek Cecho 2013-06-04, 23:15
+
Venkat Ranganathan 2013-06-05, 00:09
+
Venkat Ranganathan 2013-06-05, 03:52
+
Venkat Ranganathan 2013-06-05, 21:42
+
Jarek Cecho 2013-06-05, 21:26
+
Venkat Ranganathan 2013-06-06, 00:00
+
Venkat Ranganathan 2013-06-06, 22:55
+
Venkat Ranganathan 2013-06-07, 02:03
+
Jarek Cecho 2013-06-07, 14:29
+
Jarek Cecho 2013-06-07, 01:03
+
Venkat Ranganathan 2013-06-07, 01:53
+
Jarek Cecho 2013-06-06, 18:34
Copy link to this message
-
Re: Review Request: SQOOP-931 - Integration of Sqoop and HCatalog


> On June 6, 2013, 6:34 p.m., Jarek Cecho wrote:
> > Hi Venkat,
> > thank you very much for incorporating all my suggestions. I believe that we are almost at the end. I was again doing some testing and I've noticed few issues (some of them created by my own suggestions):
> >
> > 1) I see compilation failure
> >     [javac] /home/jarcec/apache/repos/sqoop/src/java/org/apache/sqoop/mapreduce/hcat/SqoopHCatUtilities.java:877: join(java.lang.CharSequence,java.lang.Iterable<?>) in org.apache.hadoop.util.StringUtils cannot be applied to (java.lang.String,java.lang.String[])
> >     [javac]     String argLine = StringUtils.join(",", argArray);
> >
> > I've fixed that by changing the line to String argLine = StringUtils.join(",", Arrays.asList(argArray)) to unblock the review, however proper solution is up to you :-)
> >
> > 2) We've changed the hardcoded paths to Hive and HCatalog home to SqoopOptions.getHiveHomeDefault() (or HCatalog), however those two methods actually can return null, which is causing ClassNotFoundExceptions later in the code. What about improving them in similar fashion:
> >
> >   public static String getHiveHomeDefault() {
> >     // Set this with $HIVE_HOME, but -Dhive.home can override.
> >     String hiveHome = System.getenv("HIVE_HOME", "/usr/lib/hive");
> >     return System.getProperty("hive.home", hiveHome);
> >   }

Thanks for the review

1)   I did run all the tests with hadoop100 profile but it looks like StringUtils.join(String, String[]) is a new addition.   Unfortunately, there is no @since in the javadocs :(  Sorry about that
2)  Good catch - will fix it and use the default values I was using before for these two
> On June 6, 2013, 6:34 p.m., Jarek Cecho wrote:
> > bin/configure-sqoop, line 118
> > <https://reviews.apache.org/r/10688/diff/10/?file=301227#file301227line118>
> >
> >     Nit: Add HCatalog to dependency list

Will fix
> On June 6, 2013, 6:34 p.m., Jarek Cecho wrote:
> > bin/configure-sqoop, line 118
> > <https://reviews.apache.org/r/10688/diff/10/?file=301227#file301227line118>
> >
> >     Nit: Add HCatalog to dependency list

Will fix
> On June 6, 2013, 6:34 p.m., Jarek Cecho wrote:
> > bin/configure-sqoop, line 120
> > <https://reviews.apache.org/r/10688/diff/10/?file=301227#file301227line120>
> >
> >     Rest of the Sqoop is expecting variable HADOOP_COMMON_HOME whereas the underlying hcat script is expecting HADOOP_HOME, so on BigTop this line is ending with:
> >    
> >     Hadoop not found.
> >    
> >     I was able to workaround it by adding following line before the highlighted line:
> >    
> >     export HADOOP_HOME=$HADOOP_COMMON_HOME
> >    
> >     However I'm not sure whether this is the best solution or not :-/

I think that sounds like a good fix.  Thanks for that.   Let me add it and also add a comment so that it is not accidentally removed in future
- Venkat
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/10688/#review21527
-----------------------------------------------------------
On June 6, 2013, midnight, Venkat Ranganathan wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/10688/
> -----------------------------------------------------------
>
> (Updated June 6, 2013, midnight)
>
>
> Review request for Sqoop and Jarek Cecho.
>
>
> Description
> -------
>
> This patch implements the new feature of integrating HCatalog and Sqoop.   With this feature, it is possible to import and export data between Sqoop and HCatalog tables.   The document attached to SQOOP-931 JIRA issue discusses the high level appraches.  
>
> With this integration, more fidelity can be brought to the process of moving data between enterprise data stores and hadoop ecosystem.
>
>
> Diffs
> -----
>
>   bin/configure-sqoop 61ff3f2
>   build.xml 636c103