Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Sqoop, mail # user - can we use -libjars with the Sqoop tool?


Copy link to this message
-
Re: can we use -libjars with the Sqoop tool?
abhijeet gaikwad 2013-06-08, 05:05
With the approach you are taking, see if this helps:
export HADOOP_CLASSPATH=$LIBJARS

-libjars makes sure each MR task gets these jars in classpath but for the
local machine that kick starts hadoop job does not get these jars, so the
CNFE. You need to explicitly do that as above.

A better solution:
If your edge node can submit a hadoop job to the cluster, I think this
should be easy: Install Sqoop on Edge node and run the command that Venkat
suggested.

Let me know how it goes!

Thanks,
Abhijeet
On Fri, Jun 7, 2013 at 1:41 AM, Jane Wayne <[EMAIL PROTECTED]> wrote:

> venkat,
>
> could you clarify? do you mean to not use the hadoop jar command (option
> 1)? or to use the hadoop jar command and omit -config (option 2)?
>
> option 1) hadoop --config path/to/sqoop-cluster-config-dir jar
> sqoop-1.4.3.jar org.apache.sqoop.Sqoop import -libjars ${LIBJARS} --connect
> jdbc:mysql://mysql/mydb --username myuser --password mypass --table data
> --verbose --append --warehouse-dir /sqoop
>
> option 2) hadoop jar sqoop-1.4.3.jar org.apache.sqoop.Sqoop import -libjars
> ${LIBJARS} --connect jdbc:mysql://mysql/mydb --username myuser --password
> mypass --table data --verbose --append --warehouse-dir /sqoop
>
> i need -libjars because i am trying to run sqoop from an edge node that is
> not part of the hadoop cluster. is this even possible?
>
> i posted this question on the dev mailing list, since i thought i might
> have to modify sqoop. but what i am trying to explore is this scenario.
>
> W: web server (JEE container, tomcat)
> H: hadoop cluster
> D: database server (MySQL)
>
> i need to invoke sqoop on W to import data from D to H. i read that sqoop2
> has a client API, which should have met my requirements. however, i am not
> allowed to modify the hadoop cluster (cannot install sqoop onto the hadoop
> cluster).
>
> i am opened to any approach subject to that one constraint (cannot install
> sqoop onto the hadoop cluster or put third-party libs onto the hadoop
> cluster).
>
> thanks.
>
>
>
> On Thu, Jun 6, 2013 at 11:36 AM, Venkat Ranganathan <
> [EMAIL PROTECTED]> wrote:
>
>> Don't use hadoop -config ...
>>
>> Use
>>  sqoop import -libjars ....   And I am not sure why you need libjars
>> option.   For your purpose, copy mysql connector jarfile to $SQOOP_HOME/lib
>> and just run sqoop import
>>
>>
>> Venkat
>>
>>
>> On Thu, Jun 6, 2013 at 5:56 AM, Jane Wayne <[EMAIL PROTECTED]>wrote:
>>
>>> i am running the Sqoop tool as follows.
>>>
>>> export
>>> LIBJARS=sqoop-1.4.3.jar,mysql-connector-java-5.1.25-bin.jar,ant-contrib-1.0b3.jar,ant-eclipse-1.0-jvm1.2.jar,avro-1.5.3.jar,avro-ipc-1.5.3.jar,avro-mapred-1.5.3.jar,commons-io-1.4.jar,hsqldb-1.8.0.10.jar,ivy-2.1.0.jar,jackson-core-asl-1.7.3.jar,jackson-mapper-asl-1.7.3.jar,jopt-simple-3.2.jar,paranamer-2.3.jar,snappy-java-1.0.3.2.jar
>>>
>>> hadoop --config path/to/sqoop-cluster-config-dir jar sqoop-1.4.3.jar
>>> org.apache.sqoop.Sqoop import -libjars ${LIBJARS} --connect
>>> jdbc:mysql://mysql/mydb --username myuser --password mypass --table data
>>> --verbose --append --warehouse-dir /sqoop
>>>
>>> i am getting this stack trace.
>>>
>>> java.lang.RuntimeException: java.lang.ClassNotFoundException:
>>> org.apache.sqoop.mapreduce.RawKeyTextOutputFormat
>>>         at
>>> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:996)
>>>         at
>>> org.apache.hadoop.mapreduce.JobContext.getOutputFormatClass(JobContext.java:248)
>>>         at org.apache.hadoop.mapred.Task.initialize(Task.java:501)
>>>         at org.apache.hadoop.mapred.MapTask.run(MapTask.java:306)
>>>         at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
>>>         at java.security.AccessController.doPrivileged(Native Method)
>>>         at javax.security.auth.Subject.doAs(Subject.java:396)
>>>         at
>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1127)
>>>         at org.apache.hadoop.mapred.Child.main(Child.java:264)
>>> Caused by: java.lang.ClassNotFoundException: