-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/6508/
-----------------------------------------------------------

Review request for Sqoop, Arvind Prabhakar and Bilung Lee.
Description
-------

Sqoop framework needs to support multiple job types - mainly import and export job. As we're using word "job" for different purpose in the framework already, I've decided to introduce keyword "operation" for those job types. E.g. import and export operation. I do not insist on this particular word, so feel free to suggest better one.

This patch provide all necessary changes on both client, server and repository side.
This addresses bug SQOOP-544.
    https://issues.apache.org/jira/browse/SQOOP-544
Diffs
-----

  /branches/sqoop2/client/src/main/java/org/apache/sqoop/client/shell/ShowConnectorFunction.java 1371581
  /branches/sqoop2/common/src/main/java/org/apache/sqoop/job/etl/Operation.java PRE-CREATION
  /branches/sqoop2/common/src/main/java/org/apache/sqoop/json/ConnectorBean.java 1371581
  /branches/sqoop2/common/src/main/java/org/apache/sqoop/model/MConnector.java 1371581
  /branches/sqoop2/common/src/main/java/org/apache/sqoop/model/MFramework.java 1371581
  /branches/sqoop2/common/src/main/java/org/apache/sqoop/model/MOperation.java PRE-CREATION
  /branches/sqoop2/common/src/main/java/org/apache/sqoop/model/MStringInput.java 1371581
  /branches/sqoop2/common/src/test/java/org/apache/sqoop/json/TestConnectorBean.java PRE-CREATION
  /branches/sqoop2/connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcConnector.java 1371581
  /branches/sqoop2/connector/connector-mysql-jdbc/src/main/java/org/apache/sqoop/connector/mysqljdbc/MySqlJdbcConnector.java 1371581
  /branches/sqoop2/core/src/main/java/org/apache/sqoop/connector/ConnectorHandler.java 1371581
  /branches/sqoop2/core/src/main/java/org/apache/sqoop/framework/FrameworkManager.java 1371581
  /branches/sqoop2/repository/repository-derby/src/main/java/org/apache/sqoop/repository/derby/DerbyRepositoryHandler.java 1371581
  /branches/sqoop2/repository/repository-derby/src/main/java/org/apache/sqoop/repository/derby/DerbySchemaConstants.java 1371581
  /branches/sqoop2/repository/repository-derby/src/main/java/org/apache/sqoop/repository/derby/DerbySchemaQuery.java 1371581
  /branches/sqoop2/spi/src/main/java/org/apache/sqoop/connector/spi/SqoopConnector.java 1371581

Diff: https://reviews.apache.org/r/6508/diff/
Testing
-------

* I've added new unit test for testing ConnectorBean serialization&deserialization
* I've manually verified (derby database, client side)

Expected output for "show connector -a" is:
sqoop:000> show connector -a
2 connector(s) to show:
Connector with id 1:
  Name: mysql-jdbc-connector
  Class: org.apache.sqoop.connector.mysqljdbc.MySqlJdbcConnector
  Supported operations: []
Connector with id 2:
  Name: generic-jdbc-connector
  Class: org.apache.sqoop.connector.jdbc.GenericJdbcConnector
  Supported operations: [EXPORT, IMPORT]
  Forms for operation EXPORT:
    Connection form 1:
      Name: form-connection
      Input 1:
        Name: inp-conn-jdbcdriver
        Type: STRING
        Mask: false
        Size: 128
      Input 2:
        Name: inp-conn-connectstring
        Type: STRING
        Mask: false
        Size: 128
      Input 3:
        Name: inp-conn-username
        Type: STRING
        Mask: false
        Size: 36
      Input 4:
        Name: inp-conn-password
        Type: STRING
        Mask: true
        Size: 10
      Input 5:
        Name: inp-conn-jdbc-properties
        Type: MAP
  Forms for operation IMPORT:
    Connection form 1:
      Name: form-connection
      Input 1:
        Name: inp-conn-jdbcdriver
        Type: STRING
        Mask: false
        Size: 128
      Input 2:
        Name: inp-conn-connectstring
        Type: STRING
        Mask: false
        Size: 128
      Input 3:
        Name: inp-conn-username
        Type: STRING
        Mask: false
        Size: 36
      Input 4:
        Name: inp-conn-password
        Type: STRING
        Mask: true
        Size: 10
      Input 5:
        Name: inp-conn-jdbc-properties
        Type: MAP

sqoop:000>
Thanks,

Jarek Cecho

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/6508/#review10143
-----------------------------------------------------------
-1.

I've got second thoughts and I'm planning to reorganize this little bit - I believe that it makes more sense to have one connection metadata for all job types. The reasoning here is that administrator will create connection object (for example will set JDBC Url, password and username). Operator than should be able to reuse this single connection for different jobs (even for combination of imports and exports).

I'll rework my patch and update it shortly.

Jarcec

- Jarek Cecho
On Aug. 10, 2012, 7:20 a.m., Jarek Cecho wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/6508/
> -----------------------------------------------------------
>
> (Updated Aug. 10, 2012, 7:20 a.m.)
>
>
> Review request for Sqoop, Arvind Prabhakar and Bilung Lee.
>
>
> Description
> -------
>
> Sqoop framework needs to support multiple job types - mainly import and export job. As we're using word "job" for different purpose in the framework already, I've decided to introduce keyword "operation" for those job types. E.g. import and export operation. I do not insist on this particular word, so feel free to suggest better one.
>
> This patch provide all necessary changes on both client, server and repository side.
>
>
> This addresses bug SQOOP-544.
>     https://issues.apache.org/jira/browse/SQOOP-544
>
>
> Diffs
> -----
>
>   /branches/sqoop2/client/src/main/java/org/apache/sqoop/client/shell/ShowConnectorFunction.java 1371581
>   /branches/sqoop2/common/src/main/java/org/apache/sqoop/job/etl/Operation.java PRE-CREATION
>   /branches/sqoop2/common/src/main/java/org/apache/sqoop/json/ConnectorBean.java 1371581
>   /branches/sqoop2/common/src/main/java/org/apache/sqoop/model/MConnector.java 1371581
>   /branches/sqoop2/common/src/main/java/org/apache/sqoop/model/MFramework.java 1371581
>   /branches/sqoop2/common/src/main/java/org/apache/sqoop/model/MOperation.java PRE-CREATION
>   /branches/sqoop2/common/src/main/java/org/apache/sqoop/model/MStringInput.java 1371581
>   /branches/sqoop2/common/src/test/java/org/apache/sqoop/json/TestConnectorBean.java PRE-CREATION
>   /branches/sqoop2/connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcConnector.java 1371581
>   /branches/sqoop2/connector/connector-mysql-jdbc/src/main/java/org/apache/sqoop/connector/mysqljdbc/MySqlJdbcConnector.java 1371581
>   /branches/sqoop2/core/src/main/java/org/apache/sqoop/connector/ConnectorHandler.java 1371581
>   /branches/sqoop2/core/src/main/java/org/apache/sqoop/framework/FrameworkManager.java 1371581
>   /branches/sqoop2/repository/repository-derby/src/main/java/org/apache/sqoop/repository/derby/DerbyRepositoryHandler.java 1371581
>   /branches/sqoop2/repository/repository-derby/src/main/java/org/apache/sqoop/repository/derby/DerbySchemaConstants.java 1371581
>   /branches/sqoop2/repository/repository-derby/src/main/java/org/apache/sqoop/repository/derby/DerbySchemaQuery.java 1371581
>   /branches/sqoop2/spi/src/main/java/org/apache/sqoop/connector/spi/SqoopConnector.java 1371581
>
> Diff: https://reviews.apache.org/r/6508/diff/
>
>
> Testing
> -------
>
> * I've added new unit test for testing ConnectorBean serialization&deserialization
> * I've manually verified (derby database, client side)
>
> Expected output for "show connector -a" is:
> sqoop:000> show connector -a
> 2 connector(s) to show:
> Connector with id 1:
>   Name: mysql-jdbc-connector
>   Class: org.apache.sqoop.connector.mysqljdbc.MySqlJdbcConnector
>   Supported operations: []
> Connector with id 2:
>   Name: generic-jdbc-connector
>   Class: org.apache.sqoop.connector.jdbc.GenericJdbcConnector

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/6576/
-----------------------------------------------------------

Review request for Sqoop and Bilung Lee.
Description
-------

Job covered by this JIRA:

* I've included new abstraction classes for connection and job (MConnection, MJob). Both are internally represented as List<MForm>.
* I've included support for multiple job types (Import, Export). Solution should be open to any other new job types.
* I'm forcing connector to supply only one set of Forms for each job type (e.g. support IMPORT twice is incorrect usage).

Jarcec
This addresses bug SQOOP-544.
    https://issues.apache.org/jira/browse/SQOOP-544
Diffs
-----

  /branches/sqoop2/client/src/main/java/org/apache/sqoop/client/shell/ShowConnectorFunction.java 1372400
  /branches/sqoop2/common/src/main/java/org/apache/sqoop/json/ConnectorBean.java 1372400
  /branches/sqoop2/common/src/main/java/org/apache/sqoop/model/MConnection.java PRE-CREATION
  /branches/sqoop2/common/src/main/java/org/apache/sqoop/model/MConnector.java 1372400
  /branches/sqoop2/common/src/main/java/org/apache/sqoop/model/MFramework.java 1372400
  /branches/sqoop2/common/src/main/java/org/apache/sqoop/model/MJob.java PRE-CREATION
  /branches/sqoop2/common/src/main/java/org/apache/sqoop/model/MStringInput.java 1372400
  /branches/sqoop2/common/src/main/java/org/apache/sqoop/model/ModelError.java PRE-CREATION
  /branches/sqoop2/common/src/test/java/org/apache/sqoop/json/TestConnectorBean.java PRE-CREATION
  /branches/sqoop2/common/src/test/java/org/apache/sqoop/model/TestMFramework.java PRE-CREATION
  /branches/sqoop2/connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcConnector.java 1372400
  /branches/sqoop2/connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcConnectorConstants.java 1372400
  /branches/sqoop2/connector/connector-mysql-jdbc/src/main/java/org/apache/sqoop/connector/mysqljdbc/MySqlJdbcConnector.java 1372400
  /branches/sqoop2/core/src/main/java/org/apache/sqoop/connector/ConnectorHandler.java 1372400
  /branches/sqoop2/core/src/main/java/org/apache/sqoop/framework/FrameworkManager.java 1372400
  /branches/sqoop2/repository/repository-derby/src/main/java/org/apache/sqoop/repository/derby/DerbyRepositoryHandler.java 1372400
  /branches/sqoop2/repository/repository-derby/src/main/java/org/apache/sqoop/repository/derby/DerbySchemaConstants.java 1372400
  /branches/sqoop2/repository/repository-derby/src/main/java/org/apache/sqoop/repository/derby/DerbySchemaQuery.java 1372400
  /branches/sqoop2/spi/src/main/java/org/apache/sqoop/connector/spi/SqoopConnector.java 1372400

Diff: https://reviews.apache.org/r/6576/diff/
Testing
-------

* Added new test for MFramework instance creation
* mvn test
* "Life" testing
Thanks,

Jarek Cecho

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/6576/#review10288
-----------------------------------------------------------

Ship it!
Ship It!

- Bilung Lee
On Aug. 13, 2012, 1:22 p.m., Jarek Cecho wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/6576/
> -----------------------------------------------------------
>
> (Updated Aug. 13, 2012, 1:22 p.m.)
>
>
> Review request for Sqoop and Bilung Lee.
>
>
> Description
> -------
>
> Job covered by this JIRA:
>
> * I've included new abstraction classes for connection and job (MConnection, MJob). Both are internally represented as List<MForm>.
> * I've included support for multiple job types (Import, Export). Solution should be open to any other new job types.
> * I'm forcing connector to supply only one set of Forms for each job type (e.g. support IMPORT twice is incorrect usage).
>
> Jarcec
>
>
> This addresses bug SQOOP-544.
>     https://issues.apache.org/jira/browse/SQOOP-544
>
>
> Diffs
> -----
>
>   /branches/sqoop2/client/src/main/java/org/apache/sqoop/client/shell/ShowConnectorFunction.java 1372400
>   /branches/sqoop2/common/src/main/java/org/apache/sqoop/json/ConnectorBean.java 1372400
>   /branches/sqoop2/common/src/main/java/org/apache/sqoop/model/MConnection.java PRE-CREATION
>   /branches/sqoop2/common/src/main/java/org/apache/sqoop/model/MConnector.java 1372400
>   /branches/sqoop2/common/src/main/java/org/apache/sqoop/model/MFramework.java 1372400
>   /branches/sqoop2/common/src/main/java/org/apache/sqoop/model/MJob.java PRE-CREATION
>   /branches/sqoop2/common/src/main/java/org/apache/sqoop/model/MStringInput.java 1372400
>   /branches/sqoop2/common/src/main/java/org/apache/sqoop/model/ModelError.java PRE-CREATION
>   /branches/sqoop2/common/src/test/java/org/apache/sqoop/json/TestConnectorBean.java PRE-CREATION
>   /branches/sqoop2/common/src/test/java/org/apache/sqoop/model/TestMFramework.java PRE-CREATION
>   /branches/sqoop2/connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcConnector.java 1372400
>   /branches/sqoop2/connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcConnectorConstants.java 1372400
>   /branches/sqoop2/connector/connector-mysql-jdbc/src/main/java/org/apache/sqoop/connector/mysqljdbc/MySqlJdbcConnector.java 1372400
>   /branches/sqoop2/core/src/main/java/org/apache/sqoop/connector/ConnectorHandler.java 1372400
>   /branches/sqoop2/core/src/main/java/org/apache/sqoop/framework/FrameworkManager.java 1372400
>   /branches/sqoop2/repository/repository-derby/src/main/java/org/apache/sqoop/repository/derby/DerbyRepositoryHandler.java 1372400
>   /branches/sqoop2/repository/repository-derby/src/main/java/org/apache/sqoop/repository/derby/DerbySchemaConstants.java 1372400
>   /branches/sqoop2/repository/repository-derby/src/main/java/org/apache/sqoop/repository/derby/DerbySchemaQuery.java 1372400
>   /branches/sqoop2/spi/src/main/java/org/apache/sqoop/connector/spi/SqoopConnector.java 1372400
>
> Diff: https://reviews.apache.org/r/6576/diff/
>
>
> Testing
> -------
>
> * Added new test for MFramework instance creation
> * mvn test
> * "Life" testing
>
>
> Thanks,
>
> Jarek Cecho
>
>

> On Aug. 14, 2012, 5:45 p.m., Bilung Lee wrote:
> > Ship It!

The patch on the JIRA seems to be an older one.  Could you please update it there?  Thanks!
- Bilung
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/6576/#review10288
-----------------------------------------------------------
On Aug. 13, 2012, 1:22 p.m., Jarek Cecho wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/6576/
> -----------------------------------------------------------
>
> (Updated Aug. 13, 2012, 1:22 p.m.)
>
>
> Review request for Sqoop and Bilung Lee.
>
>
> Description
> -------
>
> Job covered by this JIRA:
>
> * I've included new abstraction classes for connection and job (MConnection, MJob). Both are internally represented as List<MForm>.
> * I've included support for multiple job types (Import, Export). Solution should be open to any other new job types.
> * I'm forcing connector to supply only one set of Forms for each job type (e.g. support IMPORT twice is incorrect usage).
>
> Jarcec
>
>
> This addresses bug SQOOP-544.
>     https://issues.apache.org/jira/browse/SQOOP-544
>
>
> Diffs
> -----
>
>   /branches/sqoop2/client/src/main/java/org/apache/sqoop/client/shell/ShowConnectorFunction.java 1372400
>   /branches/sqoop2/common/src/main/java/org/apache/sqoop/json/ConnectorBean.java 1372400
>   /branches/sqoop2/common/src/main/java/org/apache/sqoop/model/MConnection.java PRE-CREATION
>   /branches/sqoop2/common/src/main/java/org/apache/sqoop/model/MConnector.java 1372400
>   /branches/sqoop2/common/src/main/java/org/apache/sqoop/model/MFramework.java 1372400
>   /branches/sqoop2/common/src/main/java/org/apache/sqoop/model/MJob.java PRE-CREATION
>   /branches/sqoop2/common/src/main/java/org/apache/sqoop/model/MStringInput.java 1372400
>   /branches/sqoop2/common/src/main/java/org/apache/sqoop/model/ModelError.java PRE-CREATION
>   /branches/sqoop2/common/src/test/java/org/apache/sqoop/json/TestConnectorBean.java PRE-CREATION
>   /branches/sqoop2/common/src/test/java/org/apache/sqoop/model/TestMFramework.java PRE-CREATION
>   /branches/sqoop2/connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcConnector.java 1372400
>   /branches/sqoop2/connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcConnectorConstants.java 1372400
>   /branches/sqoop2/connector/connector-mysql-jdbc/src/main/java/org/apache/sqoop/connector/mysqljdbc/MySqlJdbcConnector.java 1372400
>   /branches/sqoop2/core/src/main/java/org/apache/sqoop/connector/ConnectorHandler.java 1372400
>   /branches/sqoop2/core/src/main/java/org/apache/sqoop/framework/FrameworkManager.java 1372400
>   /branches/sqoop2/repository/repository-derby/src/main/java/org/apache/sqoop/repository/derby/DerbyRepositoryHandler.java 1372400
>   /branches/sqoop2/repository/repository-derby/src/main/java/org/apache/sqoop/repository/derby/DerbySchemaConstants.java 1372400
>   /branches/sqoop2/repository/repository-derby/src/main/java/org/apache/sqoop/repository/derby/DerbySchemaQuery.java 1372400
>   /branches/sqoop2/spi/src/main/java/org/apache/sqoop/connector/spi/SqoopConnector.java 1372400
>
> Diff: https://reviews.apache.org/r/6576/diff/
>
>
> Testing
> -------
>
> * Added new test for MFramework instance creation
> * mvn test
> * "Life" testing
>
>
> Thanks,
>
> Jarek Cecho
>
>

> On Aug. 14, 2012, 5:45 p.m., Bilung Lee wrote:
> > Ship It!
>
> Bilung Lee wrote:
>     The patch on the JIRA seems to be an older one.  Could you please update it there?  Thanks!

Thank you for your review sir. I've just attached updated version of the patch to JIRA. Feel free to commit it sir!
- Jarek
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/6576/#review10288
-----------------------------------------------------------
On Aug. 13, 2012, 1:22 p.m., Jarek Cecho wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/6576/
> -----------------------------------------------------------
>
> (Updated Aug. 13, 2012, 1:22 p.m.)
>
>
> Review request for Sqoop and Bilung Lee.
>
>
> Description
> -------
>
> Job covered by this JIRA:
>
> * I've included new abstraction classes for connection and job (MConnection, MJob). Both are internally represented as List<MForm>.
> * I've included support for multiple job types (Import, Export). Solution should be open to any other new job types.
> * I'm forcing connector to supply only one set of Forms for each job type (e.g. support IMPORT twice is incorrect usage).
>
> Jarcec
>
>
> This addresses bug SQOOP-544.
>     https://issues.apache.org/jira/browse/SQOOP-544
>
>
> Diffs
> -----
>
>   /branches/sqoop2/client/src/main/java/org/apache/sqoop/client/shell/ShowConnectorFunction.java 1372400
>   /branches/sqoop2/common/src/main/java/org/apache/sqoop/json/ConnectorBean.java 1372400
>   /branches/sqoop2/common/src/main/java/org/apache/sqoop/model/MConnection.java PRE-CREATION
>   /branches/sqoop2/common/src/main/java/org/apache/sqoop/model/MConnector.java 1372400
>   /branches/sqoop2/common/src/main/java/org/apache/sqoop/model/MFramework.java 1372400
>   /branches/sqoop2/common/src/main/java/org/apache/sqoop/model/MJob.java PRE-CREATION
>   /branches/sqoop2/common/src/main/java/org/apache/sqoop/model/MStringInput.java 1372400
>   /branches/sqoop2/common/src/main/java/org/apache/sqoop/model/ModelError.java PRE-CREATION
>   /branches/sqoop2/common/src/test/java/org/apache/sqoop/json/TestConnectorBean.java PRE-CREATION
>   /branches/sqoop2/common/src/test/java/org/apache/sqoop/model/TestMFramework.java PRE-CREATION
>   /branches/sqoop2/connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcConnector.java 1372400
>   /branches/sqoop2/connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcConnectorConstants.java 1372400
>   /branches/sqoop2/connector/connector-mysql-jdbc/src/main/java/org/apache/sqoop/connector/mysqljdbc/MySqlJdbcConnector.java 1372400
>   /branches/sqoop2/core/src/main/java/org/apache/sqoop/connector/ConnectorHandler.java 1372400
>   /branches/sqoop2/core/src/main/java/org/apache/sqoop/framework/FrameworkManager.java 1372400
>   /branches/sqoop2/repository/repository-derby/src/main/java/org/apache/sqoop/repository/derby/DerbyRepositoryHandler.java 1372400
>   /branches/sqoop2/repository/repository-derby/src/main/java/org/apache/sqoop/repository/derby/DerbySchemaConstants.java 1372400
>   /branches/sqoop2/repository/repository-derby/src/main/java/org/apache/sqoop/repository/derby/DerbySchemaQuery.java 1372400
>   /branches/sqoop2/spi/src/main/java/org/apache/sqoop/connector/spi/SqoopConnector.java 1372400
>
> Diff: https://reviews.apache.org/r/6576/diff/
>
>
> Testing
> -------
>
> * Added new test for MFramework instance creation
> * mvn test
> * "Life" testing
>
>
> Thanks,
>
> Jarek Cecho
>
>