Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Sqoop, mail # user - can we use -libjars with the Sqoop tool?


+
Jane Wayne 2013-06-06, 00:23
+
Venkat Ranganathan 2013-06-06, 05:54
+
Jane Wayne 2013-06-06, 12:56
Copy link to this message
-
Re: can we use -libjars with the Sqoop tool?
Venkat Ranganathan 2013-06-06, 15:36
Don't use hadoop -config ...

Use
 sqoop import -libjars ....   And I am not sure why you need libjars
option.   For your purpose, copy mysql connector jarfile to $SQOOP_HOME/lib
and just run sqoop import
Venkat
On Thu, Jun 6, 2013 at 5:56 AM, Jane Wayne <[EMAIL PROTECTED]> wrote:

> i am running the Sqoop tool as follows.
>
> export
> LIBJARS=sqoop-1.4.3.jar,mysql-connector-java-5.1.25-bin.jar,ant-contrib-1.0b3.jar,ant-eclipse-1.0-jvm1.2.jar,avro-1.5.3.jar,avro-ipc-1.5.3.jar,avro-mapred-1.5.3.jar,commons-io-1.4.jar,hsqldb-1.8.0.10.jar,ivy-2.1.0.jar,jackson-core-asl-1.7.3.jar,jackson-mapper-asl-1.7.3.jar,jopt-simple-3.2.jar,paranamer-2.3.jar,snappy-java-1.0.3.2.jar
>
> hadoop --config path/to/sqoop-cluster-config-dir jar sqoop-1.4.3.jar
> org.apache.sqoop.Sqoop import -libjars ${LIBJARS} --connect
> jdbc:mysql://mysql/mydb --username myuser --password mypass --table data
> --verbose --append --warehouse-dir /sqoop
>
> i am getting this stack trace.
>
> java.lang.RuntimeException: java.lang.ClassNotFoundException:
> org.apache.sqoop.mapreduce.RawKeyTextOutputFormat
>         at
> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:996)
>         at
> org.apache.hadoop.mapreduce.JobContext.getOutputFormatClass(JobContext.java:248)
>         at org.apache.hadoop.mapred.Task.initialize(Task.java:501)
>         at org.apache.hadoop.mapred.MapTask.run(MapTask.java:306)
>         at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at javax.security.auth.Subject.doAs(Subject.java:396)
>         at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1127)
>         at org.apache.hadoop.mapred.Child.main(Child.java:264)
> Caused by: java.lang.ClassNotFoundException:
> org.apache.sqoop.mapreduce.RawKeyTextOutputFormat
>         at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>
> when i looked at the sqoop-1.4.3.jar, clearly,
> org.apache.sqoop.mapreduce.RawKeyTextOutputFormat.class exists.
>
> any ideas what's going on?
>
>
>
>
> On Thu, Jun 6, 2013 at 1:54 AM, Venkat Ranganathan <
> [EMAIL PROTECTED]> wrote:
>
>> sqoop tools  implement the ToolRunner interface.   So, you can pass
>> -libjars to sqoop (but make sure it is the first option among the list of
>> the options
>>
>> For example, sqoop import -libjars $LIBJARS other options
>>
>> Venkat
>>
>>
>> On Wed, Jun 5, 2013 at 5:23 PM, Jane Wayne <[EMAIL PROTECTED]>wrote:
>>
>>> hi,
>>>
>>> i am wondering if i can run the org.apache.sqoop.Sqoop tool like other
>>> tools from the command line using the -libjars flag/option/parameter?
>>>
>>> i have tried the following.
>>>
>>> export
>>> LIBJARS=ant-contrib-1.0b3.jar,ant-eclipse-1.0-jvm1.2.jar,avro-1.5.3.jar,avro-ipc-1.5.3.jar,avro-mapred-1.5.3.jar,commons-io-1.4.jar,hsqldb-1.8.0.10.jar,ivy-2.1.0.jar,jackson-core-asl-1.7.3.jar,jackson-mapper-asl-1.7.3.jar,jopt-simple-3.2.jar,paranamer-2.3.jar,snappy-java-1.0.3.2.jar
>>>
>>> hadoop --config path/to/sqoop-cluster-config-dir jar sqoop-1.4.3.jar
>>> org.apache.sqoop.Sqoop -libjars ${LIBJARS} import --connect
>>> jdbc:mysql://mysql/mydb --username myuser --password mypass --table data
>>> --verbose --append --warehouse-dir /sqoop
>>>
>>> what i get is the message below.
>>>
>>> No such sqoop tool: -libjars. See 'sqoop help'.
>>>
>>> obviously, this problem has something to do with sqoop's parsing of the
>>> command line options. shouldn't sqoop allow hadoop's command line options
>>> and its own at the same time?
>>>
>>> in the meanwhile, i will try to tinker a bit with that Sqoop class to
>>> see if i can get it to work with -libjars.
>>>
>>> references that i found regarding -libjars.
>>> 1. http://grepalex.com/2013/02/25/hadoop-libjars/
>>> 2.
>>> http://blog.cloudera.com/blog/2011/01/how-to-include-third-party-libraries-in-your-map-reduce-job/
+
Jane Wayne 2013-06-06, 20:11
+
abhijeet gaikwad 2013-06-08, 05:05
+
Deepak Vohra 2013-06-06, 13:28