Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce >> mail # user >> Tasktracker Permission Issue?


Copy link to this message
-
Tasktracker Permission Issue?
I have a test environment with hadoop 1.1.1 setup with Kerberos and
yesterday I zapped my mapred.local.dir on the job and task trackers as part
of some cleanup.  When I started the task trackers back up I was unable to
run MR jobs.  This seems like a permission issue, but I can't figure out
what it would be since it auto creates everything.  I didn't make any
changes to taskcontroller.cfg or mapred-site.xml.  Below is a log from the
task tracker.

   Chris

2013-09-18 10:21:27,040 INFO org.apache.hadoop.mapred.TaskTracker:
LaunchTaskAction (registerTask): attempt_201309180916_0024_m_000002_0
task's state:UNASSIGNED
2013-09-18 10:21:27,040 INFO org.apache.hadoop.mapred.TaskTracker: Trying
to launch : attempt_201309180916_0024_m_000002_0 which needs 1 slots
2013-09-18 10:21:27,040 INFO org.apache.hadoop.mapred.TaskTracker: In
TaskLauncher, current free slots : 16 and trying to launch
attempt_201309180916_0024_m_000002_0 which needs 1 slots
2013-09-18 10:21:28,524 WARN org.apache.hadoop.mapred.TaskTracker: Error
initializing attempt_201309180916_0024_m_000002_0:
org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find
taskTracker/cpenney/jobcache/job_201309180916_0024/job.xml in any of the
configured local directories
 at
org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathToRead(LocalDirAllocator.java:429)
 at
org.apache.hadoop.fs.LocalDirAllocator.getLocalPathToRead(LocalDirAllocator.java:160)
 at
org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1341)
 at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1213)
 at org.apache.hadoop.mapred.TaskTracker$5.run(TaskTracker.java:2568)
 at java.lang.Thread.run(Thread.java:662)

2013-09-18 10:21:28,525 ERROR org.apache.hadoop.mapred.TaskStatus: Trying
to set finish time for task attempt_201309180916_0024_m_000002_0 when no
start time is set, stackTrace is : java.lang.Exception
 at org.apache.hadoop.mapred.TaskStatus.setFinishTime(TaskStatus.java:145)
 at
org.apache.hadoop.mapred.TaskTracker$TaskInProgress.kill(TaskTracker.java:3285)
 at org.apache.hadoop.mapred.TaskTracker$5.run(TaskTracker.java:2578)
 at java.lang.Thread.run(Thread.java:662)

2013-09-18 10:21:28,525 INFO org.apache.hadoop.mapred.TaskTracker:
addFreeSlot : current free slots : 16
2013-09-18 10:21:28,554 INFO org.apache.hadoop.mapred.TaskTracker:
LaunchTaskAction (registerTask): attempt_201309180916_0024_m_000002_1
task's state:UNASSIGNED
2013-09-18 10:21:28,554 INFO org.apache.hadoop.mapred.TaskTracker: Trying
to launch : attempt_201309180916_0024_m_000002_1 which needs 1 slots
2013-09-18 10:21:28,554 INFO org.apache.hadoop.mapred.TaskTracker: In
TaskLauncher, current free slots : 16 and trying to launch
attempt_201309180916_0024_m_000002_1 which needs 1 slots
2013-09-18 10:21:28,595 INFO org.apache.hadoop.mapred.TaskController:
Reading task controller config from /etc/hadoop/taskcontroller.cfg
2013-09-18 10:21:28,595 INFO org.apache.hadoop.mapred.TaskController: main
: command provided 0
2013-09-18 10:21:28,595 INFO org.apache.hadoop.mapred.TaskController: main
: user is cpenney
2013-09-18 10:21:28,595 INFO org.apache.hadoop.mapred.TaskController: Good
mapred-local-dirs are /tmp/hadoop/mapred
2013-09-18 10:21:28,595 INFO org.apache.hadoop.mapred.TaskController: Can't
open
/tmp/hadoop/mapred/taskTracker/cpenney/jobcache/job_201309180916_0024/jobToken
for output - File exists
2013-09-18 10:21:28,596 WARN org.apache.hadoop.mapred.TaskTracker:
Exception while localization java.io.IOException: Job initialization failed
(255) with output: Reading task controller config from
/etc/hadoop/taskcontroller.cfg
main : command provided 0
main : user is cpenney
Good mapred-local-dirs are /tmp/hadoop/mapred
Can't open
/tmp/hadoop/mapred/taskTracker/cpenney/jobcache/job_201309180916_0024/jobToken
for output - File exists

 at
org.apache.hadoop.mapred.LinuxTaskController.initializeJob(LinuxTaskController.java:193)
 at org.apache.hadoop.mapred.TaskTracker$4.run(TaskTracker.java:1323)
 at java.security.AccessController.doPrivileged(Native Method)
 at javax.security.auth.Subject.doAs(Subject.java:396)
 at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1136)
 at
org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1298)
 at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1213)
 at org.apache.hadoop.mapred.TaskTracker$5.run(TaskTracker.java:2568)
 at java.lang.Thread.run(Thread.java:662)
Caused by: org.apache.hadoop.util.Shell$ExitCodeException:
 at org.apache.hadoop.util.Shell.runCommand(Shell.java:255)
 at org.apache.hadoop.util.Shell.run(Shell.java:182)
 at
org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:375)
 at
org.apache.hadoop.mapred.LinuxTaskController.initializeJob(LinuxTaskController.java:186)
 ... 8 more

2013-09-18 10:21:28,596 ERROR
org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException
as:cpenney cause:java.io.IOException: Job initialization failed (255) with
output: Reading task controller config from /etc/hadoop/taskcontroller.cfg
main : command provided 0
main : user is cpenney
Good mapred-local-dirs are /tmp/hadoop/mapred
Can't open
/tmp/hadoop/mapred/taskTracker/cpenney/jobcache/job_201309180916_0024/jobToken
for output - File exists

2013-09-18 10:21:28,596 WARN org.apache.hadoop.mapred.TaskTracker: Error
initializing attempt_201309180916_0024_m_000002_1:
java.io.IOException: Job initialization failed (255) with output: Reading
task controller config from /etc/hadoop/taskcontroller.cfg
main : command provided 0
main : user is cpenney
Good mapred-local-dirs are /tmp/hadoop/mapred
Can't open
/tmp/hadoop/mapred/taskTracker/cpenney/jobcache/job_201309180916_0024/jobToken
for output - File exists

 at
org.apache.hadoop.mapred.LinuxTaskController.initializeJob(LinuxTaskController.java:193)
 at org.apache.hadoop.mapred.TaskTracker$4.run(TaskTracker.java:1323)
 at java.security.AccessController.doPrivileged(Native Meth