Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce >> mail # user >> RE: Permission problem


Copy link to this message
-
Re: Permission problem
Based on the logs your system dir is set to

> hdfs://devubuntu05:9000/data/hadoop/tmp/hadoop-mapred/mapred/system
what is your fs.default.name and hadoop.tmp.dir in core-site.xml set to?

--
Arpit Gupta
Hortonworks Inc.
http://hortonworks.com/

On Apr 30, 2013, at 7:39 AM, "Kevin Burton" <[EMAIL PROTECTED]> wrote:

>  
> Thank you.
>  
> mapred.system.dir is not set. I am guessing that it is whatever the default is. What should I set it to?
>  
> /tmp is already 777
>  
> kevin@devUbuntu05:~$ hadoop fs -ls /tmp
> Found 1 items
> drwxr-xr-x   - hdfs supergroup          0 2013-04-29 15:45 /tmp/mapred
> kevin@devUbuntu05:~$ hadoop fs -ls -d /tmp
> Found 1 items
> drwxrwxrwt   - hdfs supergroup          0 2013-04-29 15:45 /tmp
>  
> But notice that the mapred folder in the /tmp folder is 755.
> So I changed it:
>  
> kevin@devUbuntu05 $ hadoop fs -ls -d /tmp
> drwxrwxrwt   - hdfs supergroup          0 2013-04-29 15:45 /tmp
>  
> kevin@devUbuntu05 $ hadoop fs -ls -R /tmp        
> drwxr-xr-x   - mapred supergroup          0 2013-04-29 15:45 /tmp/mapred
> drwxr-xr-x   - mapred supergroup          0 2013-04-29 15:45 /tmp/mapred/system
>  
> I still get the errors in the log file:
>  
> 2013-04-30 09:35:11,609 WARN org.apache.hadoop.mapred.JobTracker: Failed to operate on mapred.system.dir (hdfs://devubuntu05:9000/data/hadoop/tmp/hadoop-mapred/mapred/system) because of permissions.
> 2013-04-30 09:35:11,609 WARN org.apache.hadoop.mapred.JobTracker: This directory should be owned by the user 'mapred (auth:SIMPLE)'
> 2013-04-30 09:35:11,609 WARN org.apache.hadoop.mapred.JobTracker: Bailing out ...
> . . . . .
> org.apache.hadoop.security.AccessControlException: Permission denied: user=mapred, access=WRITE, inode="/":hdfs:supergroup:drwxrwxr-x
> . . . . .
> Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.AccessControlException): Permission denied: user=mapred, access=WRITE, inode="/":hdfs:supergroup:drwxrwxr-x
> 2013-04-30 09:35:11,610 FATAL org.apache.hadoop.mapred.JobTracker: org.apache.hadoop.security.AccessControlException: Permission denied: user=mapred, access=WRITE, inode="/":hdfs:supergroup:drwxrwxr-x
> . . . . . .
>  
>  
> From: Arpit Gupta [mailto:[EMAIL PROTECTED]]
> Sent: Tuesday, April 30, 2013 9:25 AM
> To: [EMAIL PROTECTED]
> Subject: Re: Permission problem
>  
> what is your mapred.system.dir set to in mapred-site.xml?
>  
> By default it will write to /tmp on hdfs.
>  
> So you can do the following
>  
> create /tmp on hdfs and chmod it to 777 as user hdfs and then restart jobtracker and tasktrackers.
>  
> In case its set to /mapred/something then create /mapred and chown it to user mapred.
>  
>
> --
> Arpit Gupta
> Hortonworks Inc.
> http://hortonworks.com/
>  
> On Apr 30, 2013, at 6:36 AM, "Kevin Burton" <[EMAIL PROTECTED]> wrote:
>
>
> To further complicate the issue the log file in (/var/log/hadoop-0.20-mapreduce/hadoop-hadoop-jobtracker-devUbuntu05.log) is owned by mapred:mapred and the name of the file seems to indicate some other lineage (hadoop,hadoop). I am out of my league in understanding the permission structure for hadoop hdfs and mr. Ideas?
>  
> From: Kevin Burton [mailto:[EMAIL PROTECTED]]
> Sent: Tuesday, April 30, 2013 8:31 AM
> To: [EMAIL PROTECTED]
> Cc: 'Mohammad Tariq'
> Subject: RE: Permission problem
>  
> That is what I perceive as the problem. The hdfs file system was created with the user ‘hdfs’ owning the root (‘/’) but for some reason with a M/R job the user ‘mapred’ needs to have write permission to the root. I don’t know how to satisfy both conditions. That is one reason that I relaxed the permission to 775 so that the group would also have write permission but that didn’t seem to help.
>  
> From: Mohammad Tariq [mailto:[EMAIL PROTECTED]]
> Sent: Tuesday, April 30, 2013 8:20 AM
> To: Kevin Burton
> Subject: Re: Permission problem
>  
> user?"ls" shows "hdfs" and the log says "mapred"..
>
> Warm Regards,
> Tariq
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB