Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Sqoop >> mail # user >> can we use -libjars with the Sqoop tool?


Copy link to this message
-
Re: can we use -libjars with the Sqoop tool?
Don't use hadoop -config ...

Use
 sqoop import -libjars ....   And I am not sure why you need libjars
option.   For your purpose, copy mysql connector jarfile to $SQOOP_HOME/lib
and just run sqoop import
Venkat
On Thu, Jun 6, 2013 at 5:56 AM, Jane Wayne <[EMAIL PROTECTED]> wrote:

> i am running the Sqoop tool as follows.
>
> export
> LIBJARS=sqoop-1.4.3.jar,mysql-connector-java-5.1.25-bin.jar,ant-contrib-1.0b3.jar,ant-eclipse-1.0-jvm1.2.jar,avro-1.5.3.jar,avro-ipc-1.5.3.jar,avro-mapred-1.5.3.jar,commons-io-1.4.jar,hsqldb-1.8.0.10.jar,ivy-2.1.0.jar,jackson-core-asl-1.7.3.jar,jackson-mapper-asl-1.7.3.jar,jopt-simple-3.2.jar,paranamer-2.3.jar,snappy-java-1.0.3.2.jar
>
> hadoop --config path/to/sqoop-cluster-config-dir jar sqoop-1.4.3.jar
> org.apache.sqoop.Sqoop import -libjars ${LIBJARS} --connect
> jdbc:mysql://mysql/mydb --username myuser --password mypass --table data
> --verbose --append --warehouse-dir /sqoop
>
> i am getting this stack trace.
>
> java.lang.RuntimeException: java.lang.ClassNotFoundException:
> org.apache.sqoop.mapreduce.RawKeyTextOutputFormat
>         at
> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:996)
>         at
> org.apache.hadoop.mapreduce.JobContext.getOutputFormatClass(JobContext.java:248)
>         at org.apache.hadoop.mapred.Task.initialize(Task.java:501)
>         at org.apache.hadoop.mapred.MapTask.run(MapTask.java:306)
>         at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at javax.security.auth.Subject.doAs(Subject.java:396)
>         at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1127)
>         at org.apache.hadoop.mapred.Child.main(Child.java:264)
> Caused by: java.lang.ClassNotFoundException:
> org.apache.sqoop.mapreduce.RawKeyTextOutputFormat
>         at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>
> when i looked at the sqoop-1.4.3.jar, clearly,
> org.apache.sqoop.mapreduce.RawKeyTextOutputFormat.class exists.
>
> any ideas what's going on?
>
>
>
>
> On Thu, Jun 6, 2013 at 1:54 AM, Venkat Ranganathan <
> [EMAIL PROTECTED]> wrote:
>
>> sqoop tools  implement the ToolRunner interface.   So, you can pass
>> -libjars to sqoop (but make sure it is the first option among the list of
>> the options
>>
>> For example, sqoop import -libjars $LIBJARS other options
>>
>> Venkat
>>
>>
>> On Wed, Jun 5, 2013 at 5:23 PM, Jane Wayne <[EMAIL PROTECTED]>wrote:
>>
>>> hi,
>>>
>>> i am wondering if i can run the org.apache.sqoop.Sqoop tool like other
>>> tools from the command line using the -libjars flag/option/parameter?
>>>
>>> i have tried the following.
>>>
>>> export
>>> LIBJARS=ant-contrib-1.0b3.jar,ant-eclipse-1.0-jvm1.2.jar,avro-1.5.3.jar,avro-ipc-1.5.3.jar,avro-mapred-1.5.3.jar,commons-io-1.4.jar,hsqldb-1.8.0.10.jar,ivy-2.1.0.jar,jackson-core-asl-1.7.3.jar,jackson-mapper-asl-1.7.3.jar,jopt-simple-3.2.jar,paranamer-2.3.jar,snappy-java-1.0.3.2.jar
>>>
>>> hadoop --config path/to/sqoop-cluster-config-dir jar sqoop-1.4.3.jar
>>> org.apache.sqoop.Sqoop -libjars ${LIBJARS} import --connect
>>> jdbc:mysql://mysql/mydb --username myuser --password mypass --table data
>>> --verbose --append --warehouse-dir /sqoop
>>>
>>> what i get is the message below.
>>>
>>> No such sqoop tool: -libjars. See 'sqoop help'.
>>>
>>> obviously, this problem has something to do with sqoop's parsing of the
>>> command line options. shouldn't sqoop allow hadoop's command line options
>>> and its own at the same time?
>>>
>>> in the meanwhile, i will try to tinker a bit with that Sqoop class to
>>> see if i can get it to work with -libjars.
>>>
>>> references that i found regarding -libjars.
>>> 1. http://grepalex.com/2013/02/25/hadoop-libjars/
>>> 2.
>>> http://blog.cloudera.com/blog/2011/01/how-to-include-third-party-libraries-in-your-map-reduce-job/
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB