Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Sqoop >> mail # user >> ClassNotFoundException thrown by DBConfiguration


Copy link to this message
-
ClassNotFoundException thrown by DBConfiguration
Hi,

I encountered a ClassNotFoundException when I used DBInputFormat. I
packaged my codes with mysql-connector-java-5.1.24.jar to abc.jar. Then I
run the following command to run:

hadoop jar abc.jar a.b.c.Main

But it is crashed by the following exception:

  Notes: java.lang.RuntimeException: java.lang.RuntimeException:
java.lang.ClassNotFoundException: com.mysql.jdbc.Driver
at
org.apache.hadoop.mapreduce.lib.db.DBInputFormat.setConf(DBInputFormat.java:158)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:70)
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:130)
at org.apache.hadoop.mapred.JobClient.writeNewSplits(JobClient.java:959)
at org.apache.hadoop.mapred.JobClient.writeSplits(JobClient.java:979)
at org.apache.hadoop.mapred.JobClient.access$500(JobClient.java:170)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:891)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:844)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:416)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)
at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:844)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:481)
at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:511)
...
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at
com.microstrategy.etl.alert.feedback.FeedbackDumpTask.main(FeedbackDumpTask.java:17)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException:
com.mysql.jdbc.Driver
at
org.apache.hadoop.mapreduce.lib.db.DBInputFormat.getConnection(DBInputFormat.java:184)
at
org.apache.hadoop.mapreduce.lib.db.DBInputFormat.setConf(DBInputFormat.java:152)
... 25 more
Caused by: java.lang.ClassNotFoundException: com.mysql.jdbc.Driver
at java.net.URLClassLoader$1.run(URLClassLoader.java:217)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:205)
at java.lang.ClassLoader.loadClass(ClassLoader.java:321)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294)
at java.lang.ClassLoader.loadClass(ClassLoader.java:266)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:186)
at
org.apache.hadoop.mapreduce.lib.db.DBConfiguration.getConnection(DBConfiguration.java:143)
at
org.apache.hadoop.mapreduce.lib.db.DBInputFormat.getConnection(DBInputFormat.java:178)
... 26 more

I read the DBConfiguration.java file, and found it uses "Class.forName" to
load "com.mysql.jdbc.Driver":

   /** Returns a connection object to the DB.
   * @throws ClassNotFoundException
   * @throws SQLException */
  public Connection getConnection()
      throws ClassNotFoundException, SQLException {
    Connection connection;

    Class.forName(conf.get(DBConfiguration.DRIVER_CLASS_PROPERTY));

I think that's why my application crashed. The classloader of
DBConfiguration is sun.misc.Launcher$AppClassLoader, but
"com.mysql.jdbc.Driver" exists in the URLClassLoader created by
org.apache.hadoop.util.RunJar.
So "Class.forName" invoked by DBConfiguration can not find
"com.mysql.jdbc.Driver" in the sun.misc.Launcher$AppClassLoader.

Why does DBConfiguration not use "conf.getClassByName" to load the jdbc
driver? If DBConfiguration use the following code, it can work in such
situation.

conf.getClassByName(conf.get(DBConfiguration.DRIVER_CLASS_PROPERTY));

Is there any similar issue? Sorry that I can not find any JIRA about this
problem.

--
Thanks,
Shixiong Zhu
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB