Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Sqoop >> mail # user >> ClassNotFoundException thrown by DBConfiguration


+
Shixiong Zhu 2013-04-19, 07:02
Copy link to this message
-
Re: ClassNotFoundException thrown by DBConfiguration
Hi,

No need to pack mysql packages into your jar.
Option 1: Copy mysql jdbc jar into $HADOOP_HOME/lib. Restart hadoop
daemons.

Option 2: while executing try : *$hadoop jar your.jar a.b.c.Main -libjar
/path/mysql.jar*
On Fri, Apr 19, 2013 at 12:32 PM, Shixiong Zhu <[EMAIL PROTECTED]> wrote:

> Hi,
>
> I encountered a ClassNotFoundException when I used DBInputFormat. I
> packaged my codes with mysql-connector-java-5.1.24.jar to abc.jar. Then I
> run the following command to run:
>
> hadoop jar abc.jar a.b.c.Main
>
> But it is crashed by the following exception:
>
>   Notes: java.lang.RuntimeException: java.lang.RuntimeException:
> java.lang.ClassNotFoundException: com.mysql.jdbc.Driver
>  at
> org.apache.hadoop.mapreduce.lib.db.DBInputFormat.setConf(DBInputFormat.java:158)
> at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:70)
>  at
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:130)
> at org.apache.hadoop.mapred.JobClient.writeNewSplits(JobClient.java:959)
>  at org.apache.hadoop.mapred.JobClient.writeSplits(JobClient.java:979)
> at org.apache.hadoop.mapred.JobClient.access$500(JobClient.java:170)
>  at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:891)
> at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:844)
>  at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:416)
>  at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)
> at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:844)
>  at org.apache.hadoop.mapreduce.Job.submit(Job.java:481)
> at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:511)
>  ...
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at
> com.microstrategy.etl.alert.feedback.FeedbackDumpTask.main(FeedbackDumpTask.java:17)
>  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>  at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
>  at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
> Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException:
> com.mysql.jdbc.Driver
>  at
> org.apache.hadoop.mapreduce.lib.db.DBInputFormat.getConnection(DBInputFormat.java:184)
> at
> org.apache.hadoop.mapreduce.lib.db.DBInputFormat.setConf(DBInputFormat.java:152)
>  ... 25 more
> Caused by: java.lang.ClassNotFoundException: com.mysql.jdbc.Driver
> at java.net.URLClassLoader$1.run(URLClassLoader.java:217)
>  at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:205)
>  at java.lang.ClassLoader.loadClass(ClassLoader.java:321)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294)
>  at java.lang.ClassLoader.loadClass(ClassLoader.java:266)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:186)
>  at
> org.apache.hadoop.mapreduce.lib.db.DBConfiguration.getConnection(DBConfiguration.java:143)
> at
> org.apache.hadoop.mapreduce.lib.db.DBInputFormat.getConnection(DBInputFormat.java:178)
>  ... 26 more
>
> I read the DBConfiguration.java file, and found it uses "Class.forName" to
> load "com.mysql.jdbc.Driver":
>
>    /** Returns a connection object to the DB.
>    * @throws ClassNotFoundException
>    * @throws SQLException */
>   public Connection getConnection()
>       throws ClassNotFoundException, SQLException {
>     Connection connection;
>
>     Class.forName(conf.get(DBConfiguration.DRIVER_CLASS_PROPERTY));
>
> I think that's why my application crashed. The classloader of
> DBConfiguration is sun.misc.Launcher$AppClassLoader, but
> "com.mysql.jdbc.Driver" exists in the URLClassLoader created by
> org.apache.hadoop.util.RunJar.
> So "Class.forName" invoked by DBConfiguration can not find
> "com.mysql.jdbc.Driver" in the sun.misc.Launcher$AppClassLoader.

Regards
Vasanth kumar RJ
+
Shixiong Zhu 2013-04-19, 08:48
+
Jarek Jarcec Cecho 2013-04-19, 22:23
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB