Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Hadoop >> mail # user >> incorrect staging area path in 2.0.5


+
Pierre-Francois Laquerre 2013-08-01, 14:57
Copy link to this message
-
RE: incorrect staging area path in 2.0.5
Hi Pierre,

As per the below information, we see Job is running in local mode and trying to use the local file system for staging dir. Could you please configure 'mapreduce.framework.name'  & 'fs.defaultFS' and check.

http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/ClusterSetup.html

Thanks
Devaraj k

From: Pierre-Francois Laquerre [mailto:[EMAIL PROTECTED]]
Sent: 01 August 2013 22:57
To: [EMAIL PROTECTED]
Subject: incorrect staging area path in 2.0.5

I recently updated from 1.0.4 to 2.0.5. Since then, streaming jobs have been failing to launch due to what seems like an incorrect staging path:

# /opt/hadoop2/bin/hadoop jar /opt/hadoop2/share/hadoop/tools/lib/hadoop-streaming-2.0.5-alpha.jar -input foo -output bar -mapper baz -reducer foobar
13/08/01 10:43:50 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
13/08/01 10:43:50 WARN conf.Configuration: session.id is deprecated. Instead, use dfs.metrics.session-id
13/08/01 10:43:50 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId13/08/01 10:43:50 INFO jvm.JvmMetrics: Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
13/08/01 10:43:50 INFO mapreduce.JobSubmitter: Cleaning up the staging area file:/user/myuser1544460269/.staging/job_local1544460269_0001<file:///\\user\myuser1544460269\.staging\job_local1544460269_0001>
13/08/01 10:43:50 ERROR security.UserGroupInformation: PriviledgedActionException as:myuser (auth:SIMPLE) cause:org.apache.hadoop.util.Shell$ExitCodeException: chmod: cannot access `/user/myuser1544460269/.staging/job_local1544460269_0001': No such file or directory
13/08/01 10:43:50 ERROR streaming.StreamJob: Error Launching job : chmod: cannot access `/user/myuser1544460269/.staging/job_local1544460269_0001': No such file or directory
Streaming Command Failed!

This is for a job launched as "myuser". Given that mapreduce.jobtracker.staging.root.dir is set to /user, I would expect the staging area to be in /user/myuser/.staging/job_local$jobid. Instead, it is in /user/myuser$jobid/.staging/job_local$jobid, which fails since /user/myuser$jobid/ doesn't exist. Has the way staging.root.dir is used changed in 2.x?

Thank you,

Pierre