Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce >> mail # user >> Bug in LocalJobRunner?


Copy link to this message
-
Bug in LocalJobRunner?
Hi!

In a test I tried to run a job which is in separate jar with
MiniMRYarnCluster and got ClassNotFoundException for the Mapper class that
is in that jar.

I went a bit deeper and found possible bug in LocalJobRunner:

currently:

      this.job = new JobConf(localJobFile);

      // Job (the current object) is a Thread, so we wrap its class loader.
      if (localDistributedCacheManager.hasLocalClasspaths()) {
        setContextClassLoader(localDistributedCacheManager.makeClassLoader(
                getContextClassLoader()));
      }

should be:

      this.job = new JobConf(localJobFile);
      // Job (the current object) is a Thread, so we wrap its class loader.
      if (localDistributedCacheManager.hasLocalClasspaths()) {
        ClassLoader classLoader localDistributedCacheManager.makeClassLoader(getContextClassLoader());
        setContextClassLoader(classLoader);
        this.job.setClassLoader(classLoader);
      }

I.e. we need to set classloader for job configuration so that it can load
classes from the jar.

If the above makes sense I will file JIRA with patch, otherwise, what am I
missing?
Thank you,
Alex Baranau
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB