Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HDFS >> mail # user >> RE: Permission problem


Copy link to this message
-
Re: Permission problem
ah

this is what mapred.sytem.dir defaults to

<property>
  <name>mapred.system.dir</name>
  <value>${hadoop.tmp.dir}/mapred/system</value>
  <description>The directory where MapReduce stores control files.
  </description>
</property>
So thats why its trying to write to /data/hadoop/tmp/hadoop-mapred/mapred/system
So if you want hadoop.tmp.dir to be /data/hadoop/tmp/hadoop-${user.name}  then i would suggest that create /data/hadoop/tmp on hdfs and chmod -R 777 /data or you can remove the hadoop.tmp.dir from your configs and let it be set to the default value of

<property>
  <name>hadoop.tmp.dir</name>
  <value>/tmp/hadoop-${user.name}</value>
  <description>A base for other temporary directories.</description>
</property>

So to fix your problem you can do the above or set mapred.system.dir to /tmp/mapred/system in your mapred-site.xml.

--
Arpit Gupta
Hortonworks Inc.
http://hortonworks.com/

On Apr 30, 2013, at 7:55 AM, "Kevin Burton" <[EMAIL PROTECTED]> wrote:

> In core-site.xml I have:
>  
> <property>
>   <name>fs.default.name</name>
>   <value>hdfs://devubuntu05:9000</value>
>   <description>The name of the default file system. A URI whose scheme and authority determine the FileSystem implementation. </description>
> </property>
>  
> In hdfs-site.xml I have
>  
> <property>
>   <name>hadoop.tmp.dir</name>
>   <value>/data/hadoop/tmp/hadoop-${user.name}</value>
>   <description>Hadoop temporary folder</description>
> </property>
>  
>  
> From: Arpit Gupta [mailto:[EMAIL PROTECTED]]
> Sent: Tuesday, April 30, 2013 9:48 AM
> To: Kevin Burton
> Cc: [EMAIL PROTECTED]
> Subject: Re: Permission problem
>  
> Based on the logs your system dir is set to
>  
> hdfs://devubuntu05:9000/data/hadoop/tmp/hadoop-mapred/mapred/system
>  
>  
> what is your fs.default.name and hadoop.tmp.dir in core-site.xml set to?
>  
>  
>
> --
> Arpit Gupta
> Hortonworks Inc.
> http://hortonworks.com/
>  
> On Apr 30, 2013, at 7:39 AM, "Kevin Burton" <[EMAIL PROTECTED]> wrote:
>
>
>  
> Thank you.
>  
> mapred.system.dir is not set. I am guessing that it is whatever the default is. What should I set it to?
>  
> /tmp is already 777
>  
> kevin@devUbuntu05:~$ hadoop fs -ls /tmp
> Found 1 items
> drwxr-xr-x   - hdfs supergroup          0 2013-04-29 15:45 /tmp/mapred
> kevin@devUbuntu05:~$ hadoop fs -ls -d /tmp
> Found 1 items
> drwxrwxrwt   - hdfs supergroup          0 2013-04-29 15:45 /tmp
>  
> But notice that the mapred folder in the /tmp folder is 755.
> So I changed it:
>  
> kevin@devUbuntu05 $ hadoop fs -ls -d /tmp
> drwxrwxrwt   - hdfs supergroup          0 2013-04-29 15:45 /tmp
>  
> kevin@devUbuntu05 $ hadoop fs -ls -R /tmp        
> drwxr-xr-x   - mapred supergroup          0 2013-04-29 15:45 /tmp/mapred
> drwxr-xr-x   - mapred supergroup          0 2013-04-29 15:45 /tmp/mapred/system
>  
> I still get the errors in the log file:
>  
> 2013-04-30 09:35:11,609 WARN org.apache.hadoop.mapred.JobTracker: Failed to operate on mapred.system.dir (hdfs://devubuntu05:9000/data/hadoop/tmp/hadoop-mapred/mapred/system) because of permissions.
> 2013-04-30 09:35:11,609 WARN org.apache.hadoop.mapred.JobTracker: This directory should be owned by the user 'mapred (auth:SIMPLE)'
> 2013-04-30 09:35:11,609 WARN org.apache.hadoop.mapred.JobTracker: Bailing out ...
> . . . . .
> org.apache.hadoop.security.AccessControlException: Permission denied: user=mapred, access=WRITE, inode="/":hdfs:supergroup:drwxrwxr-x
> . . . . .
> Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.AccessControlException): Permission denied: user=mapred, access=WRITE, inode="/":hdfs:supergroup:drwxrwxr-x
> 2013-04-30 09:35:11,610 FATAL org.apache.hadoop.mapred.JobTracker: org.apache.hadoop.security.AccessControlException: Permission denied: user=mapred, access=WRITE, inode="/":hdfs:supergroup:drwxrwxr-x
> . . . . . .
>  
>  
> From: Arpit Gupta [mailto:[EMAIL PROTECTED]]
> Sent: Tuesday, April 30, 2013 9:25 AM
> To: [EMAIL PROTECTED]
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB