Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # user >> Re: Job driver and 3rd party jars


Copy link to this message
-
Re: Job driver and 3rd party jars
One alternative could be to copy the jar files to $HADOOP_HOME/lib of all
the slaves. This
had worked for me before when using external jar files.
On Fri, Mar 8, 2013 at 5:42 AM, Barak Yaish <[EMAIL PROTECTED]> wrote:

> Still doesn't work. Is this works for you? Can you upload some working
> example so I can verify I didn't miss something?
>
>
> On Fri, Mar 8, 2013 at 9:15 AM, 刘晓文 <[EMAIL PROTECTED]> wrote:
>
>> try:
>> hadoop jar .... *-Dmapreduce.task.classpath.user.precedence=true *-libjars
>> your_jar
>>
>>
>>
>> ------------------ Original ------------------
>> *From: * "Barak Yaish"<[EMAIL PROTECTED]>;
>> *Date: * Fri, Mar 8, 2013 03:06 PM
>> *To: * "user"<[EMAIL PROTECTED]>; **
>> *Subject: * Re: Job driver and 3rd party jars
>>
>> Yep, my typo, I'm using the later. I was also trying export
>> HADOOP_CLASSPATH_USER_FIRST =true and export HADOOP_CLASSPATH=myjar before
>> launching the hadoop jar, but I still getting the same exception.
>> I'm running hadoop 1.0.4.
>> On Mar 8, 2013 2:27 AM, "Harsh J" <[EMAIL PROTECTED]> wrote:
>>
>>> To be precise, did you use -libjar or -libjars? The latter is the right
>>> option.
>>>
>>> On Fri, Mar 8, 2013 at 12:18 AM, Barak Yaish <[EMAIL PROTECTED]>
>>> wrote:
>>> > Hi,
>>> >
>>> > I'm able to run M/R jobs where the mapper and reducer required to use
>>> 3rd
>>> > party jars. I'm registering those jars in -libjar while invoking the
>>> hadoop
>>> > jar command. I'm facing a strange problem, though, when the job driver
>>> > itself ( extends Configured implements Tool ) required to run such
>>> code (
>>> > for example notify some remote service upon start and end). Is there a
>>> way
>>> > to configure classpath when submitting jobs using hadoop jar? Seems
>>> like
>>> > -libjar doesn't work for this case...
>>> >
>>> > Exception in thread "main" java.lang.NoClassDefFoundError:
>>> > com/me/context/DefaultContext
>>> > at java.lang.ClassLoader.defineClass1(Native Method)
>>> > at java.lang.ClassLoader.defineClassCond(ClassLoader.java:632)
>>> > at java.lang.ClassLoader.defineClass(ClassLoader.java:616)
>>> > at
>>> > java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141)
>>> > at java.net.URLClassLoader.defineClass(URLClassLoader.java:283)
>>> > at java.net.URLClassLoader.access$000(URLClassLoader.java:58)
>>> > at java.net.URLClassLoader$1.run(URLClassLoader.java:197)
>>> > at java.security.AccessController.doPrivileged(Native Method)
>>> > at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>>> > at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
>>> > at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
>>> > at
>>> > com.peer39.bigdata.mr.pnm.PnmDataCruncher.run(PnmDataCruncher.java:50)
>>> > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
>>> > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
>>> > at com.me.mr.pnm.PnmMR.main(PnmDataCruncher.java:261)
>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>> > at
>>> >
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>> > at
>>> >
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>>> > at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
>>> > Caused by: java.lang.ClassNotFoundException:
>>> com.me.context.DefaultContext
>>> > at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>>> > at java.security.AccessController.doPrivileged(Native Method)
>>> > at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>>> > at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
>>> > at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
>>>
>>>
>>>
>>> --
>>> Harsh J
>>>
>>
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB