Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce, mail # user - Tasktracker Permission Issue?


Copy link to this message
-
Re: Tasktracker Permission Issue?
Vinod Kumar Vavilapalli 2013-09-18, 17:26
What is your config set to for mapred local dirs? And what are the permissions to those directories?

All users need executable permissions in all the paths up to the local-dir so that they can create their own directories in there. For e.g. if one of the mapred local dir is /a/b/c/mapred, then all of /a, /a/b, /a/b/c etc need to be executable by everyone - an executable permission is needed in a linux dir for someone to be able to create files/dir in some of the sub-directories.

Thanks,
+Vinod Kumar Vavilapalli
Hortonworks Inc.
http://hortonworks.com/

On Sep 18, 2013, at 7:26 AM, Christopher Penney wrote:

> I have a test environment with hadoop 1.1.1 setup with Kerberos and yesterday I zapped my mapred.local.dir on the job and task trackers as part of some cleanup.  When I started the task trackers back up I was unable to run MR jobs.  This seems like a permission issue, but I can't figure out what it would be since it auto creates everything.  I didn't make any changes to taskcontroller.cfg or mapred-site.xml.  Below is a log from the task tracker.
>
>    Chris
>
> 2013-09-18 10:21:27,040 INFO org.apache.hadoop.mapred.TaskTracker: LaunchTaskAction (registerTask): attempt_201309180916_0024_m_000002_0 task's state:UNASSIGNED
> 2013-09-18 10:21:27,040 INFO org.apache.hadoop.mapred.TaskTracker: Trying to launch : attempt_201309180916_0024_m_000002_0 which needs 1 slots
> 2013-09-18 10:21:27,040 INFO org.apache.hadoop.mapred.TaskTracker: In TaskLauncher, current free slots : 16 and trying to launch attempt_201309180916_0024_m_000002_0 which needs 1 slots
> 2013-09-18 10:21:28,524 WARN org.apache.hadoop.mapred.TaskTracker: Error initializing attempt_201309180916_0024_m_000002_0:
> org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/cpenney/jobcache/job_201309180916_0024/job.xml in any of the configured local directories
>  at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathToRead(LocalDirAllocator.java:429)
>  at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathToRead(LocalDirAllocator.java:160)
>  at org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1341)
>  at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1213)
>  at org.apache.hadoop.mapred.TaskTracker$5.run(TaskTracker.java:2568)
>  at java.lang.Thread.run(Thread.java:662)
>
> 2013-09-18 10:21:28,525 ERROR org.apache.hadoop.mapred.TaskStatus: Trying to set finish time for task attempt_201309180916_0024_m_000002_0 when no start time is set, stackTrace is : java.lang.Exception
>  at org.apache.hadoop.mapred.TaskStatus.setFinishTime(TaskStatus.java:145)
>  at org.apache.hadoop.mapred.TaskTracker$TaskInProgress.kill(TaskTracker.java:3285)
>  at org.apache.hadoop.mapred.TaskTracker$5.run(TaskTracker.java:2578)
>  at java.lang.Thread.run(Thread.java:662)
>
> 2013-09-18 10:21:28,525 INFO org.apache.hadoop.mapred.TaskTracker: addFreeSlot : current free slots : 16
> 2013-09-18 10:21:28,554 INFO org.apache.hadoop.mapred.TaskTracker: LaunchTaskAction (registerTask): attempt_201309180916_0024_m_000002_1 task's state:UNASSIGNED
> 2013-09-18 10:21:28,554 INFO org.apache.hadoop.mapred.TaskTracker: Trying to launch : attempt_201309180916_0024_m_000002_1 which needs 1 slots
> 2013-09-18 10:21:28,554 INFO org.apache.hadoop.mapred.TaskTracker: In TaskLauncher, current free slots : 16 and trying to launch attempt_201309180916_0024_m_000002_1 which needs 1 slots
> 2013-09-18 10:21:28,595 INFO org.apache.hadoop.mapred.TaskController: Reading task controller config from /etc/hadoop/taskcontroller.cfg
> 2013-09-18 10:21:28,595 INFO org.apache.hadoop.mapred.TaskController: main : command provided 0
> 2013-09-18 10:21:28,595 INFO org.apache.hadoop.mapred.TaskController: main : user is cpenney
> 2013-09-18 10:21:28,595 INFO org.apache.hadoop.mapred.TaskController: Good mapred-local-dirs are /tmp/hadoop/mapred
> 2013-09-18 10:21:28,595 INFO org.apache.hadoop.mapred.TaskController: Can't open /tmp/hadoop/mapred/taskTracker/cpenney/jobcache/job_201309180916_0024/jobToken for output - File exists
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.