Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # user >> Re: Job driver and 3rd party jars


Copy link to this message
-
Re: Job driver and 3rd party jars
One alternative could be to copy the jar files to $HADOOP_HOME/lib of all
the slaves. This
had worked for me before when using external jar files.
On Fri, Mar 8, 2013 at 5:42 AM, Barak Yaish <[EMAIL PROTECTED]> wrote:

> Still doesn't work. Is this works for you? Can you upload some working
> example so I can verify I didn't miss something?
>
>
> On Fri, Mar 8, 2013 at 9:15 AM, 刘晓文 <[EMAIL PROTECTED]> wrote:
>
>> try:
>> hadoop jar .... *-Dmapreduce.task.classpath.user.precedence=true *-libjars
>> your_jar
>>
>>
>>
>> ------------------ Original ------------------
>> *From: * "Barak Yaish"<[EMAIL PROTECTED]>;
>> *Date: * Fri, Mar 8, 2013 03:06 PM
>> *To: * "user"<[EMAIL PROTECTED]>; **
>> *Subject: * Re: Job driver and 3rd party jars
>>
>> Yep, my typo, I'm using the later. I was also trying export
>> HADOOP_CLASSPATH_USER_FIRST =true and export HADOOP_CLASSPATH=myjar before
>> launching the hadoop jar, but I still getting the same exception.
>> I'm running hadoop 1.0.4.
>> On Mar 8, 2013 2:27 AM, "Harsh J" <[EMAIL PROTECTED]> wrote:
>>
>>> To be precise, did you use -libjar or -libjars? The latter is the right
>>> option.
>>>
>>> On Fri, Mar 8, 2013 at 12:18 AM, Barak Yaish <[EMAIL PROTECTED]>
>>> wrote:
>>> > Hi,
>>> >
>>> > I'm able to run M/R jobs where the mapper and reducer required to use
>>> 3rd
>>> > party jars. I'm registering those jars in -libjar while invoking the
>>> hadoop
>>> > jar command. I'm facing a strange problem, though, when the job driver
>>> > itself ( extends Configured implements Tool ) required to run such
>>> code (
>>> > for example notify some remote service upon start and end). Is there a
>>> way
>>> > to configure classpath when submitting jobs using hadoop jar? Seems
>>> like
>>> > -libjar doesn't work for this case...
>>> >
>>> > Exception in thread "main" java.lang.NoClassDefFoundError:
>>> > com/me/context/DefaultContext
>>> > at java.lang.ClassLoader.defineClass1(Native Method)
>>> > at java.lang.ClassLoader.defineClassCond(ClassLoader.java:632)
>>> > at java.lang.ClassLoader.defineClass(ClassLoader.java:616)
>>> > at
>>> > java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141)
>>> > at java.net.URLClassLoader.defineClass(URLClassLoader.java:283)
>>> > at java.net.URLClassLoader.access$000(URLClassLoader.java:58)
>>> > at java.net.URLClassLoader$1.run(URLClassLoader.java:197)
>>> > at java.security.AccessController.doPrivileged(Native Method)
>>> > at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>>> > at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
>>> > at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
>>> > at
>>> > com.peer39.bigdata.mr.pnm.PnmDataCruncher.run(PnmDataCruncher.java:50)
>>> > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
>>> > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
>>> > at com.me.mr.pnm.PnmMR.main(PnmDataCruncher.java:261)
>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>> > at
>>> >
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>> > at
>>> >
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>>> > at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
>>> > Caused by: java.lang.ClassNotFoundException:
>>> com.me.context.DefaultContext
>>> > at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>>> > at java.security.AccessController.doPrivileged(Native Method)
>>> > at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>>> > at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
>>> > at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
>>>
>>>
>>>
>>> --
>>> Harsh J
>>>
>>
>