Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Sqoop >> mail # dev >> How to invoke extractor during IMPORT Job Submition process- Sqoop version 1.99.1 Connector


+
Kunal Sachdeva 2013-10-09, 13:01
Copy link to this message
-
Re: How to invoke extractor during IMPORT Job Submition process- Sqoop version 1.99.1 Connector
Hi Kunal,
Sqoop2 is currently under heavy development, so we do not have a "How to create connector" guide available at the moment. We will definitely have one once the interfaces will settle down.

For the time being, I'm afraid that you will need to follow existing code. If you need to call custom import extractor in your new connector, you will need to specify it in the Importer structure that is exposed by your connector. Please take a look on the Generic JDBC Connector example:

https://github.com/apache/sqoop/blob/sqoop2/connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcConnector.java#L38

Jarcec

On Wed, Oct 09, 2013 at 06:31:22PM +0530, Kunal Sachdeva wrote:
> Hi,
>
> *Subject: Want to invoke custom extractor class run method for printing
> data- MySql table's columns values data in Import job*.
>
> I am newby to Sqoop2 (*Currently working on connector version 1.99.1*) and
> want some help from you. I have developed Sqoop2 connector similar to
> generic connector 1.99.1 provided by apache, here
> http://sqoop.apache.org/docs/1.99.1/
> .
>
> In other words my connector is 99% similar to generic connector 1.99.1 but
> have different name for packages and classes.I have tested with my own
> connector  Import and Export both are working fine. But I want to print
> data during Import job submission and before it writes data to
> given/specific file, I mean when we import MySql Database table and submit
> the Import job then whatever data Sqoop2 is importing from that table, I
> want all rows,columns to be appended with my name or some other value 'abc'
> through connector.
>
> I know, Each connector in Sqoop2 extends class SqoopConnector by providing
> classes that should be called. If I need to run custom extractor, I will
> need to create custom connector that will have different extractor. On the
> client side, I will have to create connection for your new connector
> instead of the build-in Generic JDBC Connector.
>
> I have already done these above steps and My connector is working fine for
> both Import,Export jobs and I have seen result in desired location.
>
> But, as said in Import job, to append anything in table rows data before
> writing them to any file , we have to code in extractor. My extractor is
> 100% same as generic connector extractor. But when and how to invoke this
> custom conector extractor is the main thing which I do not know and I want
> to invoke this.
>
> Please help me which class and method I need to add/modify so that I can
> get invoke my connector extractor.
>
> I have followed all steps mentioned here
> http://sqoop.apache.org/docs/1.99.1/ but invoking extractor is hard thing
> for me. I do not know which value I have to give in command prompt and all.
>
> Thanks in advance.
>
> --
> Regards,
> Kunal
+
K Singh 2013-10-17, 09:44
+
Jarek Jarcec Cecho 2013-10-22, 15:35
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB