Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
MapReduce, mail # user - Bug in LocalJobRunner?


Copy link to this message
-
Bug in LocalJobRunner?
Alex Baranau 2013-03-21, 00:44
Hi!

In a test I tried to run a job which is in separate jar with
MiniMRYarnCluster and got ClassNotFoundException for the Mapper class that
is in that jar.

I went a bit deeper and found possible bug in LocalJobRunner:

currently:

      this.job = new JobConf(localJobFile);

      // Job (the current object) is a Thread, so we wrap its class loader.
      if (localDistributedCacheManager.hasLocalClasspaths()) {
        setContextClassLoader(localDistributedCacheManager.makeClassLoader(
                getContextClassLoader()));
      }

should be:

      this.job = new JobConf(localJobFile);
      // Job (the current object) is a Thread, so we wrap its class loader.
      if (localDistributedCacheManager.hasLocalClasspaths()) {
        ClassLoader classLoader localDistributedCacheManager.makeClassLoader(getContextClassLoader());
        setContextClassLoader(classLoader);
        this.job.setClassLoader(classLoader);
      }

I.e. we need to set classloader for job configuration so that it can load
classes from the jar.

If the above makes sense I will file JIRA with patch, otherwise, what am I
missing?
Thank you,
Alex Baranau
+
Alex Baranau 2013-03-22, 21:57