Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce, mail # user - Cache file conflict


Copy link to this message
-
Cache file conflict
Public Network Services 2013-08-30, 00:06
Hi...

After updating the source JARs of an application that launches a second job
while running a MR job, the following error keeps occurring:

org.apache.hadoop.mapred.InvalidJobConfException: cache file
(mapreduce.job.cache.files) scheme: "hdfs", host: "server", port: 9000,
file:
"/tmp/hadoop-yarn/staging/root/.staging/job_1367474197612_0887/libjars/Some.jar",
conflicts with cache file (mapreduce.job.cache.files)
hdfs://server:9000/tmp/hadoop-yarn/staging/root/.staging/job_1367474197612_0888/libjars/Some.jar
    at
org.apache.hadoop.mapreduce.v2.util.MRApps.parseDistributedCacheArtifacts(MRApps.java:338)
    at
org.apache.hadoop.mapreduce.v2.util.MRApps.setupDistributedCache(MRApps.java:273)
    at
org.apache.hadoop.mapred.YARNRunner.createApplicationSubmissionContext(YARNRunner.java:419)
    at org.apache.hadoop.mapred.YARNRunner.submitJob(YARNRunner.java:288)
    at
org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:394)
    at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1218)
    at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1215)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:396)
    at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1367)
    at org.apache.hadoop.mapreduce.Job.submit(Job.java:1215)
............

where job_1367474197612_0887 is the name of the initial job,
job_1367474197612_0888 is the name of the subsequent job, and Some.jar is a
JAR file specific to the application.

Any ideas as to how the above error could be eliminated?

Thanks!