Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce >> mail # dev >> DistributedCache.addFileToClassPath()


Copy link to this message
-
DistributedCache.addFileToClassPath()
Hi,
We use cdh3b2.

Recently we experience map task failure because of:
INFO [2011-02-04 18:17:36] (ExecUtil.java:261) - 2011-02-05 02:17:23,855
WARN org.apache.hadoop.mapred.TaskTracker: Error running child
INFO [2011-02-04 18:17:36] (ExecUtil.java:261) - java.io.IOException: Split
class com.carrieriq.m2m.platform.mmp2.input.FileListInputSplit not found
INFO [2011-02-04 18:17:36] (ExecUtil.java:261) -       at
org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:326)
INFO [2011-02-04 18:17:36] (ExecUtil.java:261) -       at
org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
INFO [2011-02-04 18:17:36] (ExecUtil.java:261) -       at
org.apache.hadoop.mapred.Child.main(Child.java:170)
INFO [2011-02-04 18:17:36] (ExecUtil.java:261) - Caused by:
java.lang.ClassNotFoundException:
com.carrieriq.m2m.platform.mmp2.input.FileListInputSplit
INFO [2011-02-04 18:17:36] (ExecUtil.java:261) -       at
java.net.URLClassLoader$1.run(URLClassLoader.java:202)
INFO [2011-02-04 18:17:36] (ExecUtil.java:261) -       at
java.security.AccessController.doPrivileged(Native Method)
INFO [2011-02-04 18:17:36] (ExecUtil.java:261) -       at
java.net.URLClassLoader.findClass(URLClassLoader.java:190)
INFO [2011-02-04 18:17:36] (ExecUtil.java:261) -       at
java.lang.ClassLoader.loadClass(ClassLoader.java:307)
INFO [2011-02-04 18:17:36] (ExecUtil.java:261) -       at
sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
INFO [2011-02-04 18:17:36] (ExecUtil.java:261) -       at
java.lang.ClassLoader.loadClass(ClassLoader.java:248)
INFO [2011-02-04 18:17:36] (ExecUtil.java:261) -       at
java.lang.Class.forName0(Native Method)
INFO [2011-02-04 18:17:36] (ExecUtil.java:261) -       at
java.lang.Class.forName(Class.java:247)
INFO [2011-02-04 18:17:36] (ExecUtil.java:261) -       at
org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:907)
INFO [2011-02-04 18:17:36] (ExecUtil.java:261) -       at
org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:323)
INFO [2011-02-04 18:17:36] (ExecUtil.java:261) -       ... 2 more

I found that the following config parameter was missing from the underlying
Job Conf:
mapred.job.classpath.files

We use the following code:
                Path dest = copyToDfs(jar, jobConf);
             // add URL into class path for grid based job also to ensure
flow validation can work using mmp command run
                ClassUtil.addURL(jar.toURL());
                DistributedCache.addFileToClassPath(dest, jobConf);

>From log, I verified that ClassUtil.addURL() was called.

The following API doesn't have return code, no logging:
  public static void addFileToClassPath(Path file, Configuration conf)

Please advise how I can debug this issue.

Thanks