Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
HDFS >> mail # user >> RE: Permission problem


+
Kevin Burton 2013-04-30, 14:55
+
Arpit Gupta 2013-04-30, 15:01
Copy link to this message
-
RE: Permission problem
I am not clear on what you are suggesting to create on HDFS or the local
file system. As I understand it hadoop.tmp.dir is the local file system. I
changed it so that the temporary files would be on a disk that has more
capacity then /tmp. So you are suggesting that I create /data/hadoop/tmp on
HDFS. I already have this created.

 

Found 1 items

drwxr-xr-x   - mapred supergroup          0 2013-04-29 15:45 /tmp/mapred

kevin@devUbuntu05:/etc/hadoop/conf$ hadoop fs -ls -d /tmp

Found 1 items

drwxrwxrwt   - hdfs supergroup          0 2013-04-29 15:45 /tmp

 

When you suggest that I 'chmod -R 777 /data'. You are suggesting that I open
up all the data to everyone? Isn't that a bit extreme? First /data is the
mount point for this drive and there are other uses for this drive than
hadoop so there are other folders. That is why there is /data/hadoop. As far
as hadoop is concerned:

 

kevin@devUbuntu05:/etc/hadoop/conf$ ls -l /data/hadoop/

total 12

drwxrwxr-x 4 hdfs   hadoop 4096 Apr 29 16:38 dfs

drwxrwxr-x 3 mapred hadoop 4096 Apr 29 11:33 mapred

drwxrwxrwx 3 hdfs   hadoop 4096 Apr 19 15:14 tmp

 

dfs would be where the data blocks for the hdfs file system would go, mapred
would be the folder for M/R jobs, and tmp would be temporary storage. These
are all on the local file system. Do I have to make all of this read-write
for everyone in order to get it to work?

 

From: Arpit Gupta [mailto:[EMAIL PROTECTED]]
Sent: Tuesday, April 30, 2013 10:01 AM
To: [EMAIL PROTECTED]
Subject: Re: Permission problem

 

ah

 

this is what mapred.sytem.dir defaults to

 

<property>

  <name>mapred.system.dir</name>

  <value>${hadoop.tmp.dir}/mapred/system</value>

  <description>The directory where MapReduce stores control files.

  </description>

</property>

 

 

So thats why its trying to write to
/data/hadoop/tmp/hadoop-mapred/mapred/system
<hdfs://devubuntu05:9000/data/hadoop/tmp/hadoop-mapred/mapred/system>

 

 

So if you want hadoop.tmp.dir to be /data/hadoop/tmp/hadoop-${user.name}
then i would suggest that create /data/hadoop/tmp on hdfs and chmod -R 777
/data or you can remove the hadoop.tmp.dir from your configs and let it be
set to the default value of

 

<property>

  <name>hadoop.tmp.dir</name>

  <value>/tmp/hadoop-${user.name}</value>

  <description>A base for other temporary directories.</description>

</property>

 

So to fix your problem you can do the above or set mapred.system.dir to
/tmp/mapred/system in your mapred-site.xml.
--
Arpit Gupta

Hortonworks Inc.
http://hortonworks.com/

 

On Apr 30, 2013, at 7:55 AM, "Kevin Burton" <[EMAIL PROTECTED]>
wrote:

In core-site.xml I have:

 

<property>

  <name>fs.default.name</name>

  <value> <hdfs://devubuntu05:9000%3c/value%3e>
hdfs://devubuntu05:9000</value>

  <description>The name of the default file system. A URI whose scheme and
authority determine the FileSystem implementation. </description>

</property>

 

In hdfs-site.xml I have

 

<property>

  <name>hadoop.tmp.dir</name>

  <value>/data/hadoop/tmp/hadoop-${user.name}</value>

  <description>Hadoop temporary folder</description>

</property>

 

 

From: Arpit Gupta [mailto:[EMAIL PROTECTED]]
Sent: Tuesday, April 30, 2013 9:48 AM
To: Kevin Burton
Cc: [EMAIL PROTECTED]
Subject: Re: Permission problem

 

Based on the logs your system dir is set to

 

 <hdfs://devubuntu05:9000/data/hadoop/tmp/hadoop-mapred/mapred/system>
hdfs://devubuntu05:9000/data/hadoop/tmp/hadoop-mapred/mapred/system

 

 

what is your fs.default.name and hadoop.tmp.dir in core-site.xml set to?

 

 
--
Arpit Gupta

Hortonworks Inc.
 <http://hortonworks.com/> http://hortonworks.com/

 

On Apr 30, 2013, at 7:39 AM, "Kevin Burton" <
<mailto:[EMAIL PROTECTED]> [EMAIL PROTECTED]> wrote:
 

Thank you.

 

mapred.system.dir is not set. I am guessing that it is whatever the default
is. What should I set it to?

 

/tmp is already 777

 

kevin@devUbuntu05:~$ hadoop fs -ls /tmp

Found 1 items

drwxr-xr-x   - hdfs supergroup          0 2013-04-29 15:45 /tmp/mapred

kevin@devUbuntu05:~$ hadoop fs -ls -d /tmp

Found 1 items

drwxrwxrwt   - hdfs supergroup          0 2013-04-29 15:45 /tmp

 

But notice that the mapred folder in the /tmp folder is 755.

So I changed it:

 

kevin@devUbuntu05 $ hadoop fs -ls -d /tmp

drwxrwxrwt   - hdfs supergroup          0 2013-04-29 15:45 /tmp

 

kevin@devUbuntu05 $ hadoop fs -ls -R /tmp        

drwxr-xr-x   - mapred supergroup          0 2013-04-29 15:45 /tmp/mapred

drwxr-xr-x   - mapred supergroup          0 2013-04-29 15:45
/tmp/mapred/system

 

I still get the errors in the log file:

 

2013-04-30 09:35:11,609 WARN org.apache.hadoop.mapred.JobTracker: Failed to
operate on mapred.system.dir (
<hdfs://devubuntu05:9000/data/hadoop/tmp/hadoop-mapred/mapred/system>
hdfs://devubuntu05:9000/data/hadoop/tmp/hadoop-mapred/mapred/system) because
of permissions.

2013-04-30 09:35:11,609 WARN org.apache.hadoop.mapred.JobTracker: This
directory should be owned by the user 'mapred (auth:SIMPLE)'

2013-04-30 09:35:11,609 WARN org.apache.hadoop.mapred.JobTracker: Bailing
out ...

. . . . .

org.apache.hadoop.security.AccessControlException: Permission denied:
user=mapred, access=WRITE, inode="/":hdfs:supergroup:drwxrwxr-x

. . . . .

Caused by:
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.AccessContr
olException): Permission denied: user=mapred, access=WRITE,
inode="/":hdfs:supergroup:drwxrwxr-x

2013-04-30 09:35:11,610 FATAL org.apache.hadoop.mapred.JobTracker:
org.apache.hadoop.security.AccessControlException: Permission denied:
user=mapred, access=WRITE, inode="/":hdfs:supergroup:drwxrwxr-x

. . . . . .

 

 

From: Arpit Gupta [mailto:arpit@ <http://hortonworks.com> hortonworks.com]
Sent: Tuesday, April 30, 2013 9:25 AM
To:  <mailto:[EMAIL PROTECTED]> [EMAIL PROTECTED]
Subject: Re: Permission problem

 

what is your mapred.system.dir set to in mapred-site.xml?

 

By default it will
+
Arpit Gupta 2013-04-30, 15:47
+
Mohammad Tariq 2013-04-30, 17:06
+
Kevin Burton 2013-04-30, 13:36
+
Arpit Gupta 2013-04-30, 14:25
+
Kevin Burton 2013-04-30, 14:39
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB