Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
HDFS >> mail # user >> Re: Job driver and 3rd party jars


+
Harsh J 2013-03-08, 00:27
Copy link to this message
-
Re: Job driver and 3rd party jars
Still doesn't work. Is this works for you? Can you upload some working
example so I can verify I didn't miss something?

On Fri, Mar 8, 2013 at 9:15 AM, 刘晓文 <[EMAIL PROTECTED]> wrote:

> try:
> hadoop jar .... *-Dmapreduce.task.classpath.user.precedence=true *-libjars
> your_jar
>
>
>
> ------------------ Original ------------------
> *From: * "Barak Yaish"<[EMAIL PROTECTED]>;
> *Date: * Fri, Mar 8, 2013 03:06 PM
> *To: * "user"<[EMAIL PROTECTED]>; **
> *Subject: * Re: Job driver and 3rd party jars
>
> Yep, my typo, I'm using the later. I was also trying export
> HADOOP_CLASSPATH_USER_FIRST =true and export HADOOP_CLASSPATH=myjar before
> launching the hadoop jar, but I still getting the same exception.
> I'm running hadoop 1.0.4.
> On Mar 8, 2013 2:27 AM, "Harsh J" <[EMAIL PROTECTED]> wrote:
>
>> To be precise, did you use -libjar or -libjars? The latter is the right
>> option.
>>
>> On Fri, Mar 8, 2013 at 12:18 AM, Barak Yaish <[EMAIL PROTECTED]>
>> wrote:
>> > Hi,
>> >
>> > I'm able to run M/R jobs where the mapper and reducer required to use
>> 3rd
>> > party jars. I'm registering those jars in -libjar while invoking the
>> hadoop
>> > jar command. I'm facing a strange problem, though, when the job driver
>> > itself ( extends Configured implements Tool ) required to run such code
>> (
>> > for example notify some remote service upon start and end). Is there a
>> way
>> > to configure classpath when submitting jobs using hadoop jar? Seems like
>> > -libjar doesn't work for this case...
>> >
>> > Exception in thread "main" java.lang.NoClassDefFoundError:
>> > com/me/context/DefaultContext
>> > at java.lang.ClassLoader.defineClass1(Native Method)
>> > at java.lang.ClassLoader.defineClassCond(ClassLoader.java:632)
>> > at java.lang.ClassLoader.defineClass(ClassLoader.java:616)
>> > at
>> > java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141)
>> > at java.net.URLClassLoader.defineClass(URLClassLoader.java:283)
>> > at java.net.URLClassLoader.access$000(URLClassLoader.java:58)
>> > at java.net.URLClassLoader$1.run(URLClassLoader.java:197)
>> > at java.security.AccessController.doPrivileged(Native Method)
>> > at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>> > at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
>> > at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
>> > at
>> > com.peer39.bigdata.mr.pnm.PnmDataCruncher.run(PnmDataCruncher.java:50)
>> > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
>> > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
>> > at com.me.mr.pnm.PnmMR.main(PnmDataCruncher.java:261)
>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> > at
>> >
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>> > at
>> >
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> > at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
>> > Caused by: java.lang.ClassNotFoundException:
>> com.me.context.DefaultContext
>> > at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>> > at java.security.AccessController.doPrivileged(Native Method)
>> > at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>> > at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
>> > at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
>>
>>
>>
>> --
>> Harsh J
>>
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB