Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Sqoop >> mail # user >> can we use -libjars with the Sqoop tool?


Copy link to this message
-
Re: can we use -libjars with the Sqoop tool?
venkat,

could you clarify? do you mean to not use the hadoop jar command (option
1)? or to use the hadoop jar command and omit -config (option 2)?

option 1) hadoop --config path/to/sqoop-cluster-config-dir jar
sqoop-1.4.3.jar org.apache.sqoop.Sqoop import -libjars ${LIBJARS} --connect
jdbc:mysql://mysql/mydb --username myuser --password mypass --table data
--verbose --append --warehouse-dir /sqoop

option 2) hadoop jar sqoop-1.4.3.jar org.apache.sqoop.Sqoop import -libjars
${LIBJARS} --connect jdbc:mysql://mysql/mydb --username myuser --password
mypass --table data --verbose --append --warehouse-dir /sqoop

i need -libjars because i am trying to run sqoop from an edge node that is
not part of the hadoop cluster. is this even possible?

i posted this question on the dev mailing list, since i thought i might
have to modify sqoop. but what i am trying to explore is this scenario.

W: web server (JEE container, tomcat)
H: hadoop cluster
D: database server (MySQL)

i need to invoke sqoop on W to import data from D to H. i read that sqoop2
has a client API, which should have met my requirements. however, i am not
allowed to modify the hadoop cluster (cannot install sqoop onto the hadoop
cluster).

i am opened to any approach subject to that one constraint (cannot install
sqoop onto the hadoop cluster or put third-party libs onto the hadoop
cluster).

thanks.

On Thu, Jun 6, 2013 at 11:36 AM, Venkat Ranganathan <
[EMAIL PROTECTED]> wrote:

> Don't use hadoop -config ...
>
> Use
>  sqoop import -libjars ....   And I am not sure why you need libjars
> option.   For your purpose, copy mysql connector jarfile to $SQOOP_HOME/lib
> and just run sqoop import
>
>
> Venkat
>
>
> On Thu, Jun 6, 2013 at 5:56 AM, Jane Wayne <[EMAIL PROTECTED]>wrote:
>
>> i am running the Sqoop tool as follows.
>>
>> export
>> LIBJARS=sqoop-1.4.3.jar,mysql-connector-java-5.1.25-bin.jar,ant-contrib-1.0b3.jar,ant-eclipse-1.0-jvm1.2.jar,avro-1.5.3.jar,avro-ipc-1.5.3.jar,avro-mapred-1.5.3.jar,commons-io-1.4.jar,hsqldb-1.8.0.10.jar,ivy-2.1.0.jar,jackson-core-asl-1.7.3.jar,jackson-mapper-asl-1.7.3.jar,jopt-simple-3.2.jar,paranamer-2.3.jar,snappy-java-1.0.3.2.jar
>>
>> hadoop --config path/to/sqoop-cluster-config-dir jar sqoop-1.4.3.jar
>> org.apache.sqoop.Sqoop import -libjars ${LIBJARS} --connect
>> jdbc:mysql://mysql/mydb --username myuser --password mypass --table data
>> --verbose --append --warehouse-dir /sqoop
>>
>> i am getting this stack trace.
>>
>> java.lang.RuntimeException: java.lang.ClassNotFoundException:
>> org.apache.sqoop.mapreduce.RawKeyTextOutputFormat
>>         at
>> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:996)
>>         at
>> org.apache.hadoop.mapreduce.JobContext.getOutputFormatClass(JobContext.java:248)
>>         at org.apache.hadoop.mapred.Task.initialize(Task.java:501)
>>         at org.apache.hadoop.mapred.MapTask.run(MapTask.java:306)
>>         at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
>>         at java.security.AccessController.doPrivileged(Native Method)
>>         at javax.security.auth.Subject.doAs(Subject.java:396)
>>         at
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1127)
>>         at org.apache.hadoop.mapred.Child.main(Child.java:264)
>> Caused by: java.lang.ClassNotFoundException:
>> org.apache.sqoop.mapreduce.RawKeyTextOutputFormat
>>         at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>>         at java.security.AccessController.doPrivileged(Native Method)
>>         at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>>
>> when i looked at the sqoop-1.4.3.jar, clearly,
>> org.apache.sqoop.mapreduce.RawKeyTextOutputFormat.class exists.
>>
>> any ideas what's going on?
>>
>>
>>
>>
>> On Thu, Jun 6, 2013 at 1:54 AM, Venkat Ranganathan <
>> [EMAIL PROTECTED]> wrote:
>>
>>> sqoop tools  implement the ToolRunner interface.   So, you can pass
>>> -libjars to sqoop (but make sure it is the first option among the list of
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB