Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Sqoop, mail # dev - Review Request: SQOOP-931 - Integration of Sqoop and HCatalog


+
Venkat Ranganathan 2013-04-21, 05:51
+
Venkat Ranganathan 2013-04-24, 05:13
+
Venkat Ranganathan 2013-04-29, 23:21
+
Venkat Ranganathan 2013-04-30, 06:56
+
Venkat Ranganathan 2013-05-04, 23:46
+
Jarek Cecho 2013-05-20, 13:02
+
Venkat Ranganathan 2013-05-21, 00:35
Copy link to this message
-
Re: Review Request: SQOOP-931 - Integration of Sqoop and HCatalog
Jarek Cecho 2013-05-21, 10:09


> On May 20, 2013, 1:02 p.m., Jarek Cecho wrote:
> > build.xml, line 54
> > <https://reviews.apache.org/r/10688/diff/5/?file=288026#file288026line54>
> >
> >     I'm not feeling entirely comfortable about depending on SNAPSHOTS. Is there a particular feature that we're taking advantage of in 0.6.0 that is not in 0.5.0?
>
> Venkat Ranganathan wrote:
>     No, the functionality (from the contract point of view) is even compatible with 0.4.0 I think.   I could not successfully resolve the maven repos for the earlier versions and hence I had to switch to it.   I think now I tried to build and found that only 0.11.0 is available readily at repos.maven.org.   That was the reason.  I will update and switch to 0.5.0 if that version is available in the repos.   But given that we want to have readily available Hadoop 2 and Hadoop 1 artifacts, we may have to set to 0.11.0 assuming that is the version the HCatalog team decides to publish the repositories for.

Using 0.11.0 is completely fine with me, or any other released version.
> On May 20, 2013, 1:02 p.m., Jarek Cecho wrote:
> > ivy.xml, lines 185-193
> > <https://reviews.apache.org/r/10688/diff/5/?file=288027#file288027line185>
> >
> >     Shouldn't those two dependencies be transitively propagated from HCatalog/Hive?
>
> Venkat Ranganathan wrote:
>     I had an issue building without the explicit dependency listed - may be because the repos were not having all the artifacts and the data nucleus was only available from datanucleus repository.   I will try to remove the dependency and retry.  
>     Thanks

Thank you sir, appreciated!
> On May 20, 2013, 1:02 p.m., Jarek Cecho wrote:
> > src/java/org/apache/sqoop/SqoopOptions.java, line 160
> > <https://reviews.apache.org/r/10688/diff/5/?file=288029#file288029line160>
> >
> >     Out of curiosity what the "stanza" stands for?
>
> Venkat Ranganathan wrote:
>     Stanza means paragraph :)   We used this a lot earlier in my work to describe the SQL snippets when ]we write essays to describe what we want from the database.   May be clause is a more general DB term.

Och thank you :-) I think that it's fine, I was just curious...
> On May 20, 2013, 1:02 p.m., Jarek Cecho wrote:
> > src/java/org/apache/sqoop/manager/ConnManager.java, line 197
> > <https://reviews.apache.org/r/10688/diff/5/?file=288032#file288032line197>
> >
> >     Is the timestamp mapped to String from similar reason as mentioned above with SMALLINT?
>
> Venkat Ranganathan wrote:
>     Timestamp is currently not a supported datatype in HCat (even though Hive supports it).   I will create a JIRA issue on HCat to support that now that HCatalog is a sub project of Hive.

I see, thank you for the explanation sir.
> On May 20, 2013, 1:02 p.m., Jarek Cecho wrote:
> > src/java/org/apache/sqoop/mapreduce/ExportJobBase.java, lines 202-204
> > <https://reviews.apache.org/r/10688/diff/5/?file=288034#file288034line202>
> >
> >     Similarly as in the import. Would having dedicated classes for HCatalog make sense/would be cleaner that having one class for everything and having multiple if-else statements?
>
> Venkat Ranganathan wrote:
>     Good point Jarek.  Actually I had that implementation first - but then we will not be able to support update/upsert and call by procedure would  need to be modified to handle the HCat format.   Since we were using HCat more as storage format like Avro, I decided to implement in place.  And followed similar logic for Imports as well

Thank you for your feedback. Your explanation makes complete sense to me. I believe that even the AVRO implementation is currently a bit hacky, but that will be cleaned up in Sqoop2, so I don't have any further comments.
- Jarek
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/10688/#review20756
-----------------------------------------------------------
On May 4, 2013, 11:46 p.m., Venkat Ranganathan wrote:
+
Venkat Ranganathan 2013-05-24, 23:18
+
Jarek Cecho 2013-05-28, 09:33
+
Venkat Ranganathan 2013-05-28, 20:38
+
Venkat Ranganathan 2013-05-29, 20:55
+
Venkat Ranganathan 2013-06-02, 20:33
+
Venkat Ranganathan 2013-06-03, 04:16
+
Jarek Cecho 2013-06-04, 23:15
+
Venkat Ranganathan 2013-06-05, 00:09
+
Venkat Ranganathan 2013-06-05, 03:52
+
Venkat Ranganathan 2013-06-05, 21:42
+
Jarek Cecho 2013-06-05, 21:26
+
Venkat Ranganathan 2013-06-06, 00:00
+
Venkat Ranganathan 2013-06-06, 22:55
+
Venkat Ranganathan 2013-06-07, 02:03
+
Jarek Cecho 2013-06-07, 14:29
+
Jarek Cecho 2013-06-07, 01:03
+
Venkat Ranganathan 2013-06-07, 01:53
+
Jarek Cecho 2013-06-06, 18:34
+
Venkat Ranganathan 2013-06-06, 19:07