Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce >> mail # user >> Tasktracker Permission Issue?


Copy link to this message
-
Re: Tasktracker Permission Issue?
Here is some more info.  I realized if I run the tasktracker as root it
works, but if I run it as mapred (which I assume is what I'm supposed to
do) I get the erros below.  When a job attempts running I see this under
mapred.localdir.

taskTracker:
total 4
drwxr-s--- 3 cpenney mapred 4096 Sep 18 14:53 cpenney

taskTracker/cpenney:
total 4
drwx--S--- 3 cpenney mapred 4096 Sep 18 14:53 jobcache

taskTracker/cpenney/jobcache:
total 4
drwx--S--- 4 cpenney mapred 4096 Sep 18 14:53 job_201309181359_0029

taskTracker/cpenney/jobcache/job_201309181359_0029:
total 88
drwx--S--- 3 cpenney mapred  4096 Sep 18 14:53 jars
-rw------- 1 cpenney mapred 72974 Sep 18 14:53 job.xml
-rw------- 1 cpenney mapred   230 Sep 18 14:53 jobToken
drwx--S--- 2 cpenney mapred  4096 Sep 18 14:53 work

taskTracker/cpenney/jobcache/job_201309181359_0029/jars:
total 6672
-rw------- 1 cpenney mapred   52780 Sep 18 14:53 .job.jar.crc
-rwxrwxrwx 1 cpenney mapred 6754700 Sep 18 14:53 job.jar
drwx--S--- 3 cpenney mapred    4096 Sep 18 14:53 org

taskTracker/cpenney/jobcache/job_201309181359_0029/jars/org:
total 4
drwx--S--- 3 cpenney mapred 4096 Sep 18 14:53 apache

taskTracker/cpenney/jobcache/job_201309181359_0029/jars/org/apache:
total 4
drwx--S--- 10 cpenney mapred 4096 Sep 18 14:53 pig
[snipped]

But in the log I see:

2013-09-18 14:53:59,951 WARN org.apache.hadoop.mapred.TaskTracker: Error
initializing attempt_201309181359_0029_m_000002_0:
org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find
taskTracker/cpenney/jobcache/job_201309181359_0029/job.xml in any of the
configured local directories

2013-09-18 14:53:59,952 ERROR org.apache.hadoop.mapred.TaskStatus: Trying
to set finish time for task attempt_201309181359_0029_m_000002_0 when no
start time is set, stackTrace is : java.lang.Exception

2013-09-18 14:54:00,303 WARN org.apache.hadoop.mapred.TaskTracker:
Exception while localization java.io.IOException: Job initialization failed
(255) with output: Reading task controller config from
/etc/hadoop/taskcontroller.cfg

My taskcontroller.cfg file has:

mapred.local.dir=/tmp/hadoop/mapred
hadoop.log.dir=/var/log/hadoop
mapred.tasktracker.tasks.sleeptime-before-sigkill=30
mapreduce.tasktracker.group=mapred
banned.users=mapred,hdfs
min.user.id=120

In /etc/hadoop I have:

---Sr-s--- 1 root   mapred 63382 Nov 19  2012 task-controller
-rw-r--r-- 1 root   mapred   196 Sep 18 14:30 taskcontroller.cfg
   Chris
On Wed, Sep 18, 2013 at 1:26 PM, Vinod Kumar Vavilapalli <[EMAIL PROTECTED]
> wrote:

> What is your config set to for mapred local dirs? And what are the
> permissions to those directories?
>
> All users need executable permissions in all the paths up to the local-dir
> so that they can create their own directories in there. For e.g. if one of
> the mapred local dir is /a/b/c/mapred, then all of /a, /a/b, /a/b/c etc
> need to be executable by everyone - an executable permission is needed in a
> linux dir for someone to be able to create files/dir in some of the
> sub-directories.
>
>  Thanks,
> +Vinod Kumar Vavilapalli
> Hortonworks Inc.
> http://hortonworks.com/
>
> On Sep 18, 2013, at 7:26 AM, Christopher Penney wrote:
>
> I have a test environment with hadoop 1.1.1 setup with Kerberos and
> yesterday I zapped my mapred.local.dir on the job and task trackers as part
> of some cleanup.  When I started the task trackers back up I was unable to
> run MR jobs.  This seems like a permission issue, but I can't figure out
> what it would be since it auto creates everything.  I didn't make any
> changes to taskcontroller.cfg or mapred-site.xml.  Below is a log from the
> task tracker.
>
>    Chris
>
> 2013-09-18 10:21:27,040 INFO org.apache.hadoop.mapred.TaskTracker:
> LaunchTaskAction (registerTask): attempt_201309180916_0024_m_000002_0
> task's state:UNASSIGNED
> 2013-09-18 10:21:27,040 INFO org.apache.hadoop.mapred.TaskTracker: Trying
> to launch : attempt_201309180916_0024_m_000002_0 which needs 1 slots
> 2013-09-18 10:21:27,040 INFO org.apache.hadoop.mapred.TaskTracker: In
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB