|
|
-
Re: Job driver and 3rd party jarshari 2013-03-10, 03:12
One alternative could be to copy the jar files to $HADOOP_HOME/lib of all
the slaves. This had worked for me before when using external jar files. On Fri, Mar 8, 2013 at 5:42 AM, Barak Yaish <[EMAIL PROTECTED]> wrote: > Still doesn't work. Is this works for you? Can you upload some working > example so I can verify I didn't miss something? > > > On Fri, Mar 8, 2013 at 9:15 AM, 刘晓文 <[EMAIL PROTECTED]> wrote: > >> try: >> hadoop jar .... *-Dmapreduce.task.classpath.user.precedence=true *-libjars >> your_jar >> >> >> >> ------------------ Original ------------------ >> *From: * "Barak Yaish"<[EMAIL PROTECTED]>; >> *Date: * Fri, Mar 8, 2013 03:06 PM >> *To: * "user"<[EMAIL PROTECTED]>; ** >> *Subject: * Re: Job driver and 3rd party jars >> >> Yep, my typo, I'm using the later. I was also trying export >> HADOOP_CLASSPATH_USER_FIRST =true and export HADOOP_CLASSPATH=myjar before >> launching the hadoop jar, but I still getting the same exception. >> I'm running hadoop 1.0.4. >> On Mar 8, 2013 2:27 AM, "Harsh J" <[EMAIL PROTECTED]> wrote: >> >>> To be precise, did you use -libjar or -libjars? The latter is the right >>> option. >>> >>> On Fri, Mar 8, 2013 at 12:18 AM, Barak Yaish <[EMAIL PROTECTED]> >>> wrote: >>> > Hi, >>> > >>> > I'm able to run M/R jobs where the mapper and reducer required to use >>> 3rd >>> > party jars. I'm registering those jars in -libjar while invoking the >>> hadoop >>> > jar command. I'm facing a strange problem, though, when the job driver >>> > itself ( extends Configured implements Tool ) required to run such >>> code ( >>> > for example notify some remote service upon start and end). Is there a >>> way >>> > to configure classpath when submitting jobs using hadoop jar? Seems >>> like >>> > -libjar doesn't work for this case... >>> > >>> > Exception in thread "main" java.lang.NoClassDefFoundError: >>> > com/me/context/DefaultContext >>> > at java.lang.ClassLoader.defineClass1(Native Method) >>> > at java.lang.ClassLoader.defineClassCond(ClassLoader.java:632) >>> > at java.lang.ClassLoader.defineClass(ClassLoader.java:616) >>> > at >>> > java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141) >>> > at java.net.URLClassLoader.defineClass(URLClassLoader.java:283) >>> > at java.net.URLClassLoader.access$000(URLClassLoader.java:58) >>> > at java.net.URLClassLoader$1.run(URLClassLoader.java:197) >>> > at java.security.AccessController.doPrivileged(Native Method) >>> > at java.net.URLClassLoader.findClass(URLClassLoader.java:190) >>> > at java.lang.ClassLoader.loadClass(ClassLoader.java:307) >>> > at java.lang.ClassLoader.loadClass(ClassLoader.java:248) >>> > at >>> > com.peer39.bigdata.mr.pnm.PnmDataCruncher.run(PnmDataCruncher.java:50) >>> > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) >>> > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79) >>> > at com.me.mr.pnm.PnmMR.main(PnmDataCruncher.java:261) >>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >>> > at >>> > >>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) >>> > at >>> > >>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) >>> > at java.lang.reflect.Method.invoke(Method.java:597) >>> > at org.apache.hadoop.util.RunJar.main(RunJar.java:156) >>> > Caused by: java.lang.ClassNotFoundException: >>> com.me.context.DefaultContext >>> > at java.net.URLClassLoader$1.run(URLClassLoader.java:202) >>> > at java.security.AccessController.doPrivileged(Native Method) >>> > at java.net.URLClassLoader.findClass(URLClassLoader.java:190) >>> > at java.lang.ClassLoader.loadClass(ClassLoader.java:307) >>> > at java.lang.ClassLoader.loadClass(ClassLoader.java:248) >>> >>> >>> >>> -- >>> Harsh J >>> >> > |