Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce >> mail # user >> Tasktracker Permission Issue?


Copy link to this message
-
Tasktracker Permission Issue?
I have a test environment with hadoop 1.1.1 setup with Kerberos and
yesterday I zapped my mapred.local.dir on the job and task trackers as part
of some cleanup.  When I started the task trackers back up I was unable to
run MR jobs.  This seems like a permission issue, but I can't figure out
what it would be since it auto creates everything.  I didn't make any
changes to taskcontroller.cfg or mapred-site.xml.  Below is a log from the
task tracker.

   Chris

2013-09-18 10:21:27,040 INFO org.apache.hadoop.mapred.TaskTracker:
LaunchTaskAction (registerTask): attempt_201309180916_0024_m_000002_0
task's state:UNASSIGNED
2013-09-18 10:21:27,040 INFO org.apache.hadoop.mapred.TaskTracker: Trying
to launch : attempt_201309180916_0024_m_000002_0 which needs 1 slots
2013-09-18 10:21:27,040 INFO org.apache.hadoop.mapred.TaskTracker: In
TaskLauncher, current free slots : 16 and trying to launch
attempt_201309180916_0024_m_000002_0 which needs 1 slots
2013-09-18 10:21:28,524 WARN org.apache.hadoop.mapred.TaskTracker: Error
initializing attempt_201309180916_0024_m_000002_0:
org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find
taskTracker/cpenney/jobcache/job_201309180916_0024/job.xml in any of the
configured local directories
 at
org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathToRead(LocalDirAllocator.java:429)
 at
org.apache.hadoop.fs.LocalDirAllocator.getLocalPathToRead(LocalDirAllocator.java:160)
 at
org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1341)
 at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1213)
 at org.apache.hadoop.mapred.TaskTracker$5.run(TaskTracker.java:2568)
 at java.lang.Thread.run(Thread.java:662)

2013-09-18 10:21:28,525 ERROR org.apache.hadoop.mapred.TaskStatus: Trying
to set finish time for task attempt_201309180916_0024_m_000002_0 when no
start time is set, stackTrace is : java.lang.Exception
 at org.apache.hadoop.mapred.TaskStatus.setFinishTime(TaskStatus.java:145)
 at
org.apache.hadoop.mapred.TaskTracker$TaskInProgress.kill(TaskTracker.java:3285)
 at org.apache.hadoop.mapred.TaskTracker$5.run(TaskTracker.java:2578)
 at java.lang.Thread.run(Thread.java:662)

2013-09-18 10:21:28,525 INFO org.apache.hadoop.mapred.TaskTracker:
addFreeSlot : current free slots : 16
2013-09-18 10:21:28,554 INFO org.apache.hadoop.mapred.TaskTracker:
LaunchTaskAction (registerTask): attempt_201309180916_0024_m_000002_1
task's state:UNASSIGNED
2013-09-18 10:21:28,554 INFO org.apache.hadoop.mapred.TaskTracker: Trying
to launch : attempt_201309180916_0024_m_000002_1 which needs 1 slots
2013-09-18 10:21:28,554 INFO org.apache.hadoop.mapred.TaskTracker: In
TaskLauncher, current free slots : 16 and trying to launch
attempt_201309180916_0024_m_000002_1 which needs 1 slots
2013-09-18 10:21:28,595 INFO org.apache.hadoop.mapred.TaskController:
Reading task controller config from /etc/hadoop/taskcontroller.cfg
2013-09-18 10:21:28,595 INFO org.apache.hadoop.mapred.TaskController: main
: command provided 0
2013-09-18 10:21:28,595 INFO org.apache.hadoop.mapred.TaskController: main
: user is cpenney
2013-09-18 10:21:28,595 INFO org.apache.hadoop.mapred.TaskController: Good
mapred-local-dirs are /tmp/hadoop/mapred
2013-09-18 10:21:28,595 INFO org.apache.hadoop.mapred.TaskController: Can't
open
/tmp/hadoop/mapred/taskTracker/cpenney/jobcache/job_201309180916_0024/jobToken
for output - File exists
2013-09-18 10:21:28,596 WARN org.apache.hadoop.mapred.TaskTracker:
Exception while localization java.io.IOException: Job initialization failed
(255) with output: Reading task controller config from
/etc/hadoop/taskcontroller.cfg
main : command provided 0
main : user is cpenney
Good mapred-local-dirs are /tmp/hadoop/mapred
Can't open
/tmp/hadoop/mapred/taskTracker/cpenney/jobcache/job_201309180916_0024/jobToken
for output - File exists

 at
org.apache.hadoop.mapred.LinuxTaskController.initializeJob(LinuxTaskController.java:193)
 at org.apache.hadoop.mapred.TaskTracker$4.run(TaskTracker.java:1323)
 at java.security.AccessController.doPrivileged(Native Method)
 at javax.security.auth.Subject.doAs(Subject.java:396)
 at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1136)
 at
org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1298)
 at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1213)
 at org.apache.hadoop.mapred.TaskTracker$5.run(TaskTracker.java:2568)
 at java.lang.Thread.run(Thread.java:662)
Caused by: org.apache.hadoop.util.Shell$ExitCodeException:
 at org.apache.hadoop.util.Shell.runCommand(Shell.java:255)
 at org.apache.hadoop.util.Shell.run(Shell.java:182)
 at
org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:375)
 at
org.apache.hadoop.mapred.LinuxTaskController.initializeJob(LinuxTaskController.java:186)
 ... 8 more

2013-09-18 10:21:28,596 ERROR
org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException
as:cpenney cause:java.io.IOException: Job initialization failed (255) with
output: Reading task controller config from /etc/hadoop/taskcontroller.cfg
main : command provided 0
main : user is cpenney
Good mapred-local-dirs are /tmp/hadoop/mapred
Can't open
/tmp/hadoop/mapred/taskTracker/cpenney/jobcache/job_201309180916_0024/jobToken
for output - File exists

2013-09-18 10:21:28,596 WARN org.apache.hadoop.mapred.TaskTracker: Error
initializing attempt_201309180916_0024_m_000002_1:
java.io.IOException: Job initialization failed (255) with output: Reading
task controller config from /etc/hadoop/taskcontroller.cfg
main : command provided 0
main : user is cpenney
Good mapred-local-dirs are /tmp/hadoop/mapred
Can't open
/tmp/hadoop/mapred/taskTracker/cpenney/jobcache/job_201309180916_0024/jobToken
for output - File exists

 at
org.apache.hadoop.mapred.LinuxTaskController.initializeJob(LinuxTaskController.java:193)
 at org.apache.hadoop.mapred.TaskTracker$4.run(TaskTracker.java:1323)
 at java.security.AccessController.doPrivileged(Native Meth
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB