Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Sqoop >> mail # dev >> Review Request: Fix for SQOOP-937


Copy link to this message
-
Re: Review Request: Fix for SQOOP-937


> On March 15, 2013, 4:38 a.m., Jarek Cecho wrote:
> > Hi Venkat,
> > thank you very much for cleaning this up. What about taking the idea even further by skipping entire class generation and compilation in this case? I think that some of the classes requires valid class name, but we can put dummy class to the source code directly right? There shouldn't be need to generate and compile it every time. What do you think?
> >
> > Jarcec

Good point Jarcec.   The reason I did not go the full step was there were a bunch of places we expect the jar (even if I say no jar file by returning null from ORM generator), the loadJar files used in a few places expect non-null strings.   I initially went down that path and decided against it as this has a record of why we generated this.

I think your optimization makes sense (keep it generated in the src tree).   That should be good to do and may be we can then get rid of the loadJar calls if the Jar file is null.
Thanks
Venkat
- Venkat
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/9909/#review17947
-----------------------------------------------------------
On March 13, 2013, 7:18 p.m., Venkat Ranganathan wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/9909/
> -----------------------------------------------------------
>
> (Updated March 13, 2013, 7:18 p.m.)
>
>
> Review request for Sqoop and Jarek Cecho.
>
>
> Description
> -------
>
> SQOOP generates an ORM file that represents a record in a table for a
> given connector implememntation.  The generated class has methods to
> read field values off ResultSet, set bind values in PreparedStatements,
> handle LOB objects in a DB specific way (that the connector represents),
> parse input fields etc.
>
> This file is then compiled and archived using jar and used with the SQOOP job.
>
> This ORM instance is generated in all cases, and used to make sure that
> the data is read and processed in a more or less uniform way.
>
> Unfortunately, this generated class is not used by a class of connectors which
> manage the reading and processing of records themselves.   In essence, the
> whole ORM class is unusable in these instances and is simply ignored.  These
> are typically "direct" connectors which use a DB specific highspeed path.
>
> The generation of the ORM class in these cases causes confusion to users.
>
> This patch tries to solve this by
>
>   1)  Providing a capability for the connection managers to declare that
> they don't depend on the ORM jar file.
>   2) Generating a dummy ORM jar file with explicit message during generation
> so that
>      a)  users are aware that the class generated is a dummy one
>      b)  there is a record that this jar file was generated explicitly
>      c)  we don't have to change a whole lot of the codebase to disable
>          the generaration and loading of the jar file.
>
>
> This patch also adds one test.
>
>
> Diffs
> -----
>
>   src/java/org/apache/sqoop/manager/ConnManager.java 1b32dc9
>   src/java/org/apache/sqoop/manager/DirectNetezzaManager.java 0a1e605
>   src/java/org/apache/sqoop/orm/ClassWriter.java 136982c
>   src/java/org/apache/sqoop/tool/CodeGenTool.java 8a4aa42
>   src/test/com/cloudera/sqoop/orm/TestClassWriter.java 3b77571
>
> Diff: https://reviews.apache.org/r/9909/diff/
>
>
> Testing
> -------
>
> Added one tests.  All unit tests and check style tests passed with no new checkstyle issues
>
>
> Thanks,
>
> Venkat Ranganathan
>
>