Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce >> mail # dev >> DistributedCache.addFileToClassPath()


Copy link to this message
-
DistributedCache.addFileToClassPath()
Hi,
We use cdh3b2.

Recently we experience map task failure because of:
INFO [2011-02-04 18:17:36] (ExecUtil.java:261) - 2011-02-05 02:17:23,855
WARN org.apache.hadoop.mapred.TaskTracker: Error running child
INFO [2011-02-04 18:17:36] (ExecUtil.java:261) - java.io.IOException: Split
class com.carrieriq.m2m.platform.mmp2.input.FileListInputSplit not found
INFO [2011-02-04 18:17:36] (ExecUtil.java:261) -       at
org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:326)
INFO [2011-02-04 18:17:36] (ExecUtil.java:261) -       at
org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
INFO [2011-02-04 18:17:36] (ExecUtil.java:261) -       at
org.apache.hadoop.mapred.Child.main(Child.java:170)
INFO [2011-02-04 18:17:36] (ExecUtil.java:261) - Caused by:
java.lang.ClassNotFoundException:
com.carrieriq.m2m.platform.mmp2.input.FileListInputSplit
INFO [2011-02-04 18:17:36] (ExecUtil.java:261) -       at
java.net.URLClassLoader$1.run(URLClassLoader.java:202)
INFO [2011-02-04 18:17:36] (ExecUtil.java:261) -       at
java.security.AccessController.doPrivileged(Native Method)
INFO [2011-02-04 18:17:36] (ExecUtil.java:261) -       at
java.net.URLClassLoader.findClass(URLClassLoader.java:190)
INFO [2011-02-04 18:17:36] (ExecUtil.java:261) -       at
java.lang.ClassLoader.loadClass(ClassLoader.java:307)
INFO [2011-02-04 18:17:36] (ExecUtil.java:261) -       at
sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
INFO [2011-02-04 18:17:36] (ExecUtil.java:261) -       at
java.lang.ClassLoader.loadClass(ClassLoader.java:248)
INFO [2011-02-04 18:17:36] (ExecUtil.java:261) -       at
java.lang.Class.forName0(Native Method)
INFO [2011-02-04 18:17:36] (ExecUtil.java:261) -       at
java.lang.Class.forName(Class.java:247)
INFO [2011-02-04 18:17:36] (ExecUtil.java:261) -       at
org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:907)
INFO [2011-02-04 18:17:36] (ExecUtil.java:261) -       at
org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:323)
INFO [2011-02-04 18:17:36] (ExecUtil.java:261) -       ... 2 more

I found that the following config parameter was missing from the underlying
Job Conf:
mapred.job.classpath.files

We use the following code:
                Path dest = copyToDfs(jar, jobConf);
             // add URL into class path for grid based job also to ensure
flow validation can work using mmp command run
                ClassUtil.addURL(jar.toURL());
                DistributedCache.addFileToClassPath(dest, jobConf);

>From log, I verified that ClassUtil.addURL() was called.

The following API doesn't have return code, no logging:
  public static void addFileToClassPath(Path file, Configuration conf)

Please advise how I can debug this issue.

Thanks
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB