Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HDFS, mail # user - RE: Permission problem


Copy link to this message
-
Re: Permission problem
Arpit Gupta 2013-04-30, 15:01
ah

this is what mapred.sytem.dir defaults to

<property>
  <name>mapred.system.dir</name>
  <value>${hadoop.tmp.dir}/mapred/system</value>
  <description>The directory where MapReduce stores control files.
  </description>
</property>
So thats why its trying to write to /data/hadoop/tmp/hadoop-mapred/mapred/system
So if you want hadoop.tmp.dir to be /data/hadoop/tmp/hadoop-${user.name}  then i would suggest that create /data/hadoop/tmp on hdfs and chmod -R 777 /data or you can remove the hadoop.tmp.dir from your configs and let it be set to the default value of

<property>
  <name>hadoop.tmp.dir</name>
  <value>/tmp/hadoop-${user.name}</value>
  <description>A base for other temporary directories.</description>
</property>

So to fix your problem you can do the above or set mapred.system.dir to /tmp/mapred/system in your mapred-site.xml.

--
Arpit Gupta
Hortonworks Inc.
http://hortonworks.com/

On Apr 30, 2013, at 7:55 AM, "Kevin Burton" <[EMAIL PROTECTED]> wrote:

> In core-site.xml I have:
>  
> <property>
>   <name>fs.default.name</name>
>   <value>hdfs://devubuntu05:9000</value>
>   <description>The name of the default file system. A URI whose scheme and authority determine the FileSystem implementation. </description>
> </property>
>  
> In hdfs-site.xml I have
>  
> <property>
>   <name>hadoop.tmp.dir</name>
>   <value>/data/hadoop/tmp/hadoop-${user.name}</value>
>   <description>Hadoop temporary folder</description>
> </property>
>  
>  
> From: Arpit Gupta [mailto:[EMAIL PROTECTED]]
> Sent: Tuesday, April 30, 2013 9:48 AM
> To: Kevin Burton
> Cc: [EMAIL PROTECTED]
> Subject: Re: Permission problem
>  
> Based on the logs your system dir is set to
>  
> hdfs://devubuntu05:9000/data/hadoop/tmp/hadoop-mapred/mapred/system
>  
>  
> what is your fs.default.name and hadoop.tmp.dir in core-site.xml set to?
>  
>  
>
> --
> Arpit Gupta
> Hortonworks Inc.
> http://hortonworks.com/
>  
> On Apr 30, 2013, at 7:39 AM, "Kevin Burton" <[EMAIL PROTECTED]> wrote:
>
>
>  
> Thank you.
>  
> mapred.system.dir is not set. I am guessing that it is whatever the default is. What should I set it to?
>  
> /tmp is already 777
>  
> kevin@devUbuntu05:~$ hadoop fs -ls /tmp
> Found 1 items
> drwxr-xr-x   - hdfs supergroup          0 2013-04-29 15:45 /tmp/mapred
> kevin@devUbuntu05:~$ hadoop fs -ls -d /tmp
> Found 1 items
> drwxrwxrwt   - hdfs supergroup          0 2013-04-29 15:45 /tmp
>  
> But notice that the mapred folder in the /tmp folder is 755.
> So I changed it:
>  
> kevin@devUbuntu05 $ hadoop fs -ls -d /tmp
> drwxrwxrwt   - hdfs supergroup          0 2013-04-29 15:45 /tmp
>  
> kevin@devUbuntu05 $ hadoop fs -ls -R /tmp        
> drwxr-xr-x   - mapred supergroup          0 2013-04-29 15:45 /tmp/mapred
> drwxr-xr-x   - mapred supergroup          0 2013-04-29 15:45 /tmp/mapred/system
>  
> I still get the errors in the log file:
>  
> 2013-04-30 09:35:11,609 WARN org.apache.hadoop.mapred.JobTracker: Failed to operate on mapred.system.dir (hdfs://devubuntu05:9000/data/hadoop/tmp/hadoop-mapred/mapred/system) because of permissions.
> 2013-04-30 09:35:11,609 WARN org.apache.hadoop.mapred.JobTracker: This directory should be owned by the user 'mapred (auth:SIMPLE)'
> 2013-04-30 09:35:11,609 WARN org.apache.hadoop.mapred.JobTracker: Bailing out ...
> . . . . .
> org.apache.hadoop.security.AccessControlException: Permission denied: user=mapred, access=WRITE, inode="/":hdfs:supergroup:drwxrwxr-x
> . . . . .
> Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.AccessControlException): Permission denied: user=mapred, access=WRITE, inode="/":hdfs:supergroup:drwxrwxr-x
> 2013-04-30 09:35:11,610 FATAL org.apache.hadoop.mapred.JobTracker: org.apache.hadoop.security.AccessControlException: Permission denied: user=mapred, access=WRITE, inode="/":hdfs:supergroup:drwxrwxr-x
> . . . . . .
>  
>  
> From: Arpit Gupta [mailto:[EMAIL PROTECTED]]
> Sent: Tuesday, April 30, 2013 9:25 AM
> To: [EMAIL PROTECTED]