Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Sqoop >> mail # user >> can we use -libjars with the Sqoop tool?


Copy link to this message
-
Re: can we use -libjars with the Sqoop tool?
i am running the Sqoop tool as follows.

export
LIBJARS=sqoop-1.4.3.jar,mysql-connector-java-5.1.25-bin.jar,ant-contrib-1.0b3.jar,ant-eclipse-1.0-jvm1.2.jar,avro-1.5.3.jar,avro-ipc-1.5.3.jar,avro-mapred-1.5.3.jar,commons-io-1.4.jar,hsqldb-1.8.0.10.jar,ivy-2.1.0.jar,jackson-core-asl-1.7.3.jar,jackson-mapper-asl-1.7.3.jar,jopt-simple-3.2.jar,paranamer-2.3.jar,snappy-java-1.0.3.2.jar

hadoop --config path/to/sqoop-cluster-config-dir jar sqoop-1.4.3.jar
org.apache.sqoop.Sqoop import -libjars ${LIBJARS} --connect
jdbc:mysql://mysql/mydb --username myuser --password mypass --table data
--verbose --append --warehouse-dir /sqoop

i am getting this stack trace.

java.lang.RuntimeException: java.lang.ClassNotFoundException:
org.apache.sqoop.mapreduce.RawKeyTextOutputFormat
        at
org.apache.hadoop.conf.Configuration.getClass(Configuration.java:996)
        at
org.apache.hadoop.mapreduce.JobContext.getOutputFormatClass(JobContext.java:248)
        at org.apache.hadoop.mapred.Task.initialize(Task.java:501)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:306)
        at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:396)
        at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1127)
        at org.apache.hadoop.mapred.Child.main(Child.java:264)
Caused by: java.lang.ClassNotFoundException:
org.apache.sqoop.mapreduce.RawKeyTextOutputFormat
        at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:190)

when i looked at the sqoop-1.4.3.jar, clearly,
org.apache.sqoop.mapreduce.RawKeyTextOutputFormat.class exists.

any ideas what's going on?
On Thu, Jun 6, 2013 at 1:54 AM, Venkat Ranganathan <
[EMAIL PROTECTED]> wrote:

> sqoop tools  implement the ToolRunner interface.   So, you can pass
> -libjars to sqoop (but make sure it is the first option among the list of
> the options
>
> For example, sqoop import -libjars $LIBJARS other options
>
> Venkat
>
>
> On Wed, Jun 5, 2013 at 5:23 PM, Jane Wayne <[EMAIL PROTECTED]>wrote:
>
>> hi,
>>
>> i am wondering if i can run the org.apache.sqoop.Sqoop tool like other
>> tools from the command line using the -libjars flag/option/parameter?
>>
>> i have tried the following.
>>
>> export
>> LIBJARS=ant-contrib-1.0b3.jar,ant-eclipse-1.0-jvm1.2.jar,avro-1.5.3.jar,avro-ipc-1.5.3.jar,avro-mapred-1.5.3.jar,commons-io-1.4.jar,hsqldb-1.8.0.10.jar,ivy-2.1.0.jar,jackson-core-asl-1.7.3.jar,jackson-mapper-asl-1.7.3.jar,jopt-simple-3.2.jar,paranamer-2.3.jar,snappy-java-1.0.3.2.jar
>>
>> hadoop --config path/to/sqoop-cluster-config-dir jar sqoop-1.4.3.jar
>> org.apache.sqoop.Sqoop -libjars ${LIBJARS} import --connect
>> jdbc:mysql://mysql/mydb --username myuser --password mypass --table data
>> --verbose --append --warehouse-dir /sqoop
>>
>> what i get is the message below.
>>
>> No such sqoop tool: -libjars. See 'sqoop help'.
>>
>> obviously, this problem has something to do with sqoop's parsing of the
>> command line options. shouldn't sqoop allow hadoop's command line options
>> and its own at the same time?
>>
>> in the meanwhile, i will try to tinker a bit with that Sqoop class to see
>> if i can get it to work with -libjars.
>>
>> references that i found regarding -libjars.
>> 1. http://grepalex.com/2013/02/25/hadoop-libjars/
>> 2.
>> http://blog.cloudera.com/blog/2011/01/how-to-include-third-party-libraries-in-your-map-reduce-job/
>>
>> thanks,
>>
>
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB