Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
HDFS >> mail # user >> RE: Permission problem


Copy link to this message
-
RE: Permission problem
In core-site.xml I have:

 

<property>

  <name>fs.default.name</name>

  <value>hdfs://devubuntu05:9000</value>

  <description>The name of the default file system. A URI whose scheme and
authority determine the FileSystem implementation. </description>

</property>

 

In hdfs-site.xml I have

 

<property>

  <name>hadoop.tmp.dir</name>

  <value>/data/hadoop/tmp/hadoop-${user.name}</value>

  <description>Hadoop temporary folder</description>

</property>

 

 

From: Arpit Gupta [mailto:[EMAIL PROTECTED]]
Sent: Tuesday, April 30, 2013 9:48 AM
To: Kevin Burton
Cc: [EMAIL PROTECTED]
Subject: Re: Permission problem

 

Based on the logs your system dir is set to

 

 <hdfs://devubuntu05:9000/data/hadoop/tmp/hadoop-mapred/mapred/system>
hdfs://devubuntu05:9000/data/hadoop/tmp/hadoop-mapred/mapred/system

 

 

what is your fs.default.name and hadoop.tmp.dir in core-site.xml set to?

 

 
--
Arpit Gupta

Hortonworks Inc.
http://hortonworks.com/

 

On Apr 30, 2013, at 7:39 AM, "Kevin Burton" <[EMAIL PROTECTED]>
wrote:

 

Thank you.

 

mapred.system.dir is not set. I am guessing that it is whatever the default
is. What should I set it to?

 

/tmp is already 777

 

kevin@devUbuntu05:~$ hadoop fs -ls /tmp

Found 1 items

drwxr-xr-x   - hdfs supergroup          0 2013-04-29 15:45 /tmp/mapred

kevin@devUbuntu05:~$ hadoop fs -ls -d /tmp

Found 1 items

drwxrwxrwt   - hdfs supergroup          0 2013-04-29 15:45 /tmp

 

But notice that the mapred folder in the /tmp folder is 755.

So I changed it:

 

kevin@devUbuntu05 $ hadoop fs -ls -d /tmp

drwxrwxrwt   - hdfs supergroup          0 2013-04-29 15:45 /tmp

 

kevin@devUbuntu05 $ hadoop fs -ls -R /tmp        

drwxr-xr-x   - mapred supergroup          0 2013-04-29 15:45 /tmp/mapred

drwxr-xr-x   - mapred supergroup          0 2013-04-29 15:45
/tmp/mapred/system

 

I still get the errors in the log file:

 

2013-04-30 09:35:11,609 WARN org.apache.hadoop.mapred.JobTracker: Failed to
operate on mapred.system.dir (
<hdfs://devubuntu05:9000/data/hadoop/tmp/hadoop-mapred/mapred/system>
hdfs://devubuntu05:9000/data/hadoop/tmp/hadoop-mapred/mapred/system) because
of permissions.

2013-04-30 09:35:11,609 WARN org.apache.hadoop.mapred.JobTracker: This
directory should be owned by the user 'mapred (auth:SIMPLE)'

2013-04-30 09:35:11,609 WARN org.apache.hadoop.mapred.JobTracker: Bailing
out ...

. . . . .

org.apache.hadoop.security.AccessControlException: Permission denied:
user=mapred, access=WRITE, inode="/":hdfs:supergroup:drwxrwxr-x

. . . . .

Caused by:
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.AccessContr
olException): Permission denied: user=mapred, access=WRITE,
inode="/":hdfs:supergroup:drwxrwxr-x

2013-04-30 09:35:11,610 FATAL org.apache.hadoop.mapred.JobTracker:
org.apache.hadoop.security.AccessControlException: Permission denied:
user=mapred, access=WRITE, inode="/":hdfs:supergroup:drwxrwxr-x

. . . . . .

 

 

From: Arpit Gupta [mailto:[EMAIL PROTECTED]]
Sent: Tuesday, April 30, 2013 9:25 AM
To: [EMAIL PROTECTED]
Subject: Re: Permission problem

 

what is your mapred.system.dir set to in mapred-site.xml?

 

By default it will write to /tmp on hdfs.

 

So you can do the following

 

create /tmp on hdfs and chmod it to 777 as user hdfs and then restart
jobtracker and tasktrackers.

 

In case its set to /mapred/something then create /mapred and chown it to
user mapred.

 
--
Arpit Gupta

Hortonworks Inc.
 <http://hortonworks.com/> http://hortonworks.com/

 

On Apr 30, 2013, at 6:36 AM, "Kevin Burton" <
<mailto:[EMAIL PROTECTED]> [EMAIL PROTECTED]> wrote:
To further complicate the issue the log file in
(/var/log/hadoop-0.20-mapreduce/hadoop-hadoop-jobtracker-devUbuntu05.log) is
owned by mapred:mapred and the name of the file seems to indicate some other
lineage (hadoop,hadoop). I am out of my league in understanding the
permission structure for hadoop hdfs and mr. Ideas?

 

From: Kevin Burton [mailto:rkevinburton@ <http://charter.net> charter.net]
Sent: Tuesday, April 30, 2013 8:31 AM
To:  <mailto:[EMAIL PROTECTED]> [EMAIL PROTECTED]
Cc: 'Mohammad Tariq'
Subject: RE: Permission problem

 

That is what I perceive as the problem. The hdfs file system was created
with the user 'hdfs' owning the root ('/') but for some reason with a M/R
job the user 'mapred' needs to have write permission to the root. I don't
know how to satisfy both conditions. That is one reason that I relaxed the
permission to 775 so that the group would also have write permission but
that didn't seem to help.

 

From: Mohammad Tariq [ <mailto:[EMAIL PROTECTED]>
mailto:[EMAIL PROTECTED]]
Sent: Tuesday, April 30, 2013 8:20 AM
To: Kevin Burton
Subject: Re: Permission problem

 

user?"ls" shows "hdfs" and the log says "mapred"..
Warm Regards,

Tariq

 <https://mtariq.jux.com/> https://mtariq.jux.com/

 <http://cloudfront.blogspot.com> cloudfront.blogspot.com

 

On Tue, Apr 30, 2013 at 6:22 PM, Kevin Burton <
<mailto:[EMAIL PROTECTED]> [EMAIL PROTECTED]> wrote:

I have relaxed it even further so now it is 775

 

 <mailto:kevin@devUbuntu05:/var/log/hadoop-0.20-mapreduce$>
kevin@devUbuntu05:/var/log/hadoop-0.20-mapreduce$ hadoop fs -ls -d /

Found 1 items

drwxrwxr-x   - hdfs supergroup          0 2013-04-29 15:43 /

 

But I still get this error:

 

2013-04-30 07:43:02,520 FATAL org.apache.hadoop.mapred.JobTracker:
org.apache.hadoop.security.AccessControlException: Permission denied:
user=mapred, access=WRITE, inode="/":hdfs:supergroup:drwxrwxr-x

 

 

From: Mohammad Tariq [mailto: <mailto:[EMAIL PROTECTED]>
[EMAIL PROTECTED]]
Sent: Monday, April 29, 2013 5:10 PM
To:  <mailto:[EMAIL PROTECTED]> [EMAIL PROTECTED]
Subject: Re: Incompartible cluserIDS

 

make it 755.
Warm Regards,

Tariq

 <https://mtariq.jux.com/> https://mtariq.jux.com/

 <http://cloudfront.blogspot.com> cloudfront.blogspot.com

 

 

 
+
Arpit Gupta 2013-04-30, 15:01
+
Kevin Burton 2013-04-30, 15:22
+
Arpit Gupta 2013-04-30, 15:47
+
Mohammad Tariq 2013-04-30, 17:06
+
Kevin Burton 2013-04-30, 13:36
+
Arpit Gupta 2013-04-30, 14:25
+
Kevin Burton 2013-04-30, 14:39
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB