Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
HDFS >> mail # user >> Re: Job driver and 3rd party jars


Copy link to this message
-
Re: Job driver and 3rd party jars
To be precise, did you use -libjar or -libjars? The latter is the right option.

On Fri, Mar 8, 2013 at 12:18 AM, Barak Yaish <[EMAIL PROTECTED]> wrote:
> Hi,
>
> I'm able to run M/R jobs where the mapper and reducer required to use 3rd
> party jars. I'm registering those jars in -libjar while invoking the hadoop
> jar command. I'm facing a strange problem, though, when the job driver
> itself ( extends Configured implements Tool ) required to run such code (
> for example notify some remote service upon start and end). Is there a way
> to configure classpath when submitting jobs using hadoop jar? Seems like
> -libjar doesn't work for this case...
>
> Exception in thread "main" java.lang.NoClassDefFoundError:
> com/me/context/DefaultContext
>         at java.lang.ClassLoader.defineClass1(Native Method)
>         at java.lang.ClassLoader.defineClassCond(ClassLoader.java:632)
>         at java.lang.ClassLoader.defineClass(ClassLoader.java:616)
>         at
> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141)
>         at java.net.URLClassLoader.defineClass(URLClassLoader.java:283)
>         at java.net.URLClassLoader.access$000(URLClassLoader.java:58)
>         at java.net.URLClassLoader$1.run(URLClassLoader.java:197)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>         at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
>         at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
>         at
> com.peer39.bigdata.mr.pnm.PnmDataCruncher.run(PnmDataCruncher.java:50)
>         at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
>         at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
>         at com.me.mr.pnm.PnmMR.main(PnmDataCruncher.java:261)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>         at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>         at java.lang.reflect.Method.invoke(Method.java:597)
>         at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
> Caused by: java.lang.ClassNotFoundException: com.me.context.DefaultContext
>         at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>         at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
>         at java.lang.ClassLoader.loadClass(ClassLoader.java:248)

--
Harsh J
+
Barak Yaish 2013-03-08, 10:42
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB