Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HDFS, mail # user - RE: Permission problem


Copy link to this message
-
RE: Permission problem
Kevin Burton 2013-04-30, 14:39


Thank you.

 

mapred.system.dir is not set. I am guessing that it is whatever the default
is. What should I set it to?

 

/tmp is already 777

 

kevin@devUbuntu05:~$ hadoop fs -ls /tmp

Found 1 items

drwxr-xr-x   - hdfs supergroup          0 2013-04-29 15:45 /tmp/mapred

kevin@devUbuntu05:~$ hadoop fs -ls -d /tmp

Found 1 items

drwxrwxrwt   - hdfs supergroup          0 2013-04-29 15:45 /tmp

 

But notice that the mapred folder in the /tmp folder is 755.

So I changed it:

 

kevin@devUbuntu05 $ hadoop fs -ls -d /tmp

drwxrwxrwt   - hdfs supergroup          0 2013-04-29 15:45 /tmp

 

kevin@devUbuntu05 $ hadoop fs -ls -R /tmp        

drwxr-xr-x   - mapred supergroup          0 2013-04-29 15:45 /tmp/mapred

drwxr-xr-x   - mapred supergroup          0 2013-04-29 15:45
/tmp/mapred/system

 

I still get the errors in the log file:

 

2013-04-30 09:35:11,609 WARN org.apache.hadoop.mapred.JobTracker: Failed to
operate on mapred.system.dir
(hdfs://devubuntu05:9000/data/hadoop/tmp/hadoop-mapred/mapred/system)
because of permissions.

2013-04-30 09:35:11,609 WARN org.apache.hadoop.mapred.JobTracker: This
directory should be owned by the user 'mapred (auth:SIMPLE)'

2013-04-30 09:35:11,609 WARN org.apache.hadoop.mapred.JobTracker: Bailing
out ...

. . . . .

org.apache.hadoop.security.AccessControlException: Permission denied:
user=mapred, access=WRITE, inode="/":hdfs:supergroup:drwxrwxr-x

. . . . .

Caused by:
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.AccessContr
olException): Permission denied: user=mapred, access=WRITE,
inode="/":hdfs:supergroup:drwxrwxr-x

2013-04-30 09:35:11,610 FATAL org.apache.hadoop.mapred.JobTracker:
org.apache.hadoop.security.AccessControlException: Permission denied:
user=mapred, access=WRITE, inode="/":hdfs:supergroup:drwxrwxr-x

. . . . . .

 

 

From: Arpit Gupta [mailto:[EMAIL PROTECTED]]
Sent: Tuesday, April 30, 2013 9:25 AM
To: [EMAIL PROTECTED]
Subject: Re: Permission problem

 

what is your mapred.system.dir set to in mapred-site.xml?

 

By default it will write to /tmp on hdfs.

 

So you can do the following

 

create /tmp on hdfs and chmod it to 777 as user hdfs and then restart
jobtracker and tasktrackers.

 

In case its set to /mapred/something then create /mapred and chown it to
user mapred.

 
--
Arpit Gupta

Hortonworks Inc.
http://hortonworks.com/

 

On Apr 30, 2013, at 6:36 AM, "Kevin Burton" <[EMAIL PROTECTED]>
wrote:

To further complicate the issue the log file in
(/var/log/hadoop-0.20-mapreduce/hadoop-hadoop-jobtracker-devUbuntu05.log) is
owned by mapred:mapred and the name of the file seems to indicate some other
lineage (hadoop,hadoop). I am out of my league in understanding the
permission structure for hadoop hdfs and mr. Ideas?

 

From: Kevin Burton [mailto:rkevinburton@ <http://charter.net> charter.net]
Sent: Tuesday, April 30, 2013 8:31 AM
To:  <mailto:[EMAIL PROTECTED]> [EMAIL PROTECTED]
Cc: 'Mohammad Tariq'
Subject: RE: Permission problem

 

That is what I perceive as the problem. The hdfs file system was created
with the user 'hdfs' owning the root ('/') but for some reason with a M/R
job the user 'mapred' needs to have write permission to the root. I don't
know how to satisfy both conditions. That is one reason that I relaxed the
permission to 775 so that the group would also have write permission but
that didn't seem to help.

 

From: Mohammad Tariq [ <mailto:[EMAIL PROTECTED]>
mailto:[EMAIL PROTECTED]]
Sent: Tuesday, April 30, 2013 8:20 AM
To: Kevin Burton
Subject: Re: Permission problem

 

user?"ls" shows "hdfs" and the log says "mapred"..
Warm Regards,

Tariq

 <https://mtariq.jux.com/> https://mtariq.jux.com/

 <http://cloudfront.blogspot.com> cloudfront.blogspot.com

 

On Tue, Apr 30, 2013 at 6:22 PM, Kevin Burton <
<mailto:[EMAIL PROTECTED]> [EMAIL PROTECTED]> wrote:

I have relaxed it even further so now it is 775

 

 <mailto:kevin@devUbuntu05:/var/log/hadoop-0.20-mapreduce$>
kevin@devUbuntu05:/var/log/hadoop-0.20-mapreduce$ hadoop fs -ls -d /

Found 1 items

drwxrwxr-x   - hdfs supergroup          0 2013-04-29 15:43 /

 

But I still get this error:

 

2013-04-30 07:43:02,520 FATAL org.apache.hadoop.mapred.JobTracker:
org.apache.hadoop.security.AccessControlException: Permission denied:
user=mapred, access=WRITE, inode="/":hdfs:supergroup:drwxrwxr-x

 

 

From: Mohammad Tariq [mailto: <mailto:[EMAIL PROTECTED]>
[EMAIL PROTECTED]]
Sent: Monday, April 29, 2013 5:10 PM
To:  <mailto:[EMAIL PROTECTED]> [EMAIL PROTECTED]
Subject: Re: Incompartible cluserIDS

 

make it 755.
Warm Regards,

Tariq

 <https://mtariq.jux.com/> https://mtariq.jux.com/

 <http://cloudfront.blogspot.com> cloudfront.blogspot.com