Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Sqoop >> mail # dev >> How to invoke extractor during IMPORT Job Submition process- Sqoop version 1.99.1 Connector


Copy link to this message
-
Re: How to invoke extractor during IMPORT Job Submition process- Sqoop version 1.99.1 Connector
Hi Kunal,
the extractor is connector driver rather then user given. As a result you can't specify which extractor will be used as this is specified by the connector. You will have to create your own connector if you need to call custom extractor.

Jarcec

On Thu, Oct 17, 2013 at 03:14:16PM +0530, K Singh wrote:
> Hi ,
>
> Thanks for clarification.
>
> I have already followed below URL to make my own generic connector :
>
> https://github.com/apache/sqoop/blob/sqoop2/connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcConnector.java#L38
>
>
> But the requirement is, once I have my own connector which is working fine,
> I want to invoke Extractor during IMPORT job. The built in generic
> connector also not calling Extractor. So Is there any way to invoke
> extractor or how and when Extractor is being called?
>
> In simple words, If we already developed/built generic connector and
> working fine then How can Import job submission/I call or invoke extractor
> class ( e.q. GenericJdbcImportExtractor.class,) during IMPORT process. I
> want to just call this class default method.
>
> Please guide me for this. Is there any additional command line arguments or
> something like which we can use to invoke/call extractor  because by
> default it is not being called in IMPORT job. Like we have for validation:
>
> *Validation arguments More
> Details<http://archive.cloudera.com/cdh4/cdh/4/sqoop-1.4.3-cdh4.4.0/SqoopUserGuide.html#validation>
> *
> Argument Description --validate Enable validation of data copied, supports
> single table copy only. --validator <class-name> Specify validator class to
> use. --validation-threshold <class-name> Specify validation threshold class
> to use. +--validation-failurehandler <class-name >+ Specify validation
> failure handler class to use.
>
>
> So is there anything for extractor as well?
>
> Thanks.
>
>
>
>
> On Fri, Oct 11, 2013 at 9:13 PM, Jarek Jarcec Cecho <[EMAIL PROTECTED]>wrote:
>
> > Hi Kunal,
> > Sqoop2 is currently under heavy development, so we do not have a "How to
> > create connector" guide available at the moment. We will definitely have
> > one once the interfaces will settle down.
> >
> > For the time being, I'm afraid that you will need to follow existing code.
> > If you need to call custom import extractor in your new connector, you will
> > need to specify it in the Importer structure that is exposed by your
> > connector. Please take a look on the Generic JDBC Connector example:
> >
> >
> > https://github.com/apache/sqoop/blob/sqoop2/connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcConnector.java#L38
> >
> > Jarcec
> >
> > On Wed, Oct 09, 2013 at 06:31:22PM +0530, Kunal Sachdeva wrote:
> > > Hi,
> > >
> > > *Subject: Want to invoke custom extractor class run method for printing
> > > data- MySql table's columns values data in Import job*.
> > >
> > > I am newby to Sqoop2 (*Currently working on connector version 1.99.1*)
> > and
> > > want some help from you. I have developed Sqoop2 connector similar to
> > > generic connector 1.99.1 provided by apache, here
> > > http://sqoop.apache.org/docs/1.99.1/
> > > .
> > >
> > > In other words my connector is 99% similar to generic connector 1.99.1
> > but
> > > have different name for packages and classes.I have tested with my own
> > > connector  Import and Export both are working fine. But I want to print
> > > data during Import job submission and before it writes data to
> > > given/specific file, I mean when we import MySql Database table and
> > submit
> > > the Import job then whatever data Sqoop2 is importing from that table, I
> > > want all rows,columns to be appended with my name or some other value
> > 'abc'
> > > through connector.
> > >
> > > I know, Each connector in Sqoop2 extends class SqoopConnector by
> > providing
> > > classes that should be called. If I need to run custom extractor, I will
> > > need to create custom connector that will have different extractor. On
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB