Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
MapReduce >> mail # user >> DFS Permissions on Hadoop 2.x


+
Prashant Kommireddi 2013-06-18, 17:54
+
Prashant Kommireddi 2013-06-18, 19:04
+
Chris Nauroth 2013-06-18, 20:28
+
Prashant Kommireddi 2013-06-18, 22:24
+
Prashant Kommireddi 2013-06-19, 00:48
+
Harsh J 2013-06-19, 03:40
Copy link to this message
-
Re: DFS Permissions on Hadoop 2.x
Thanks Chuan for the details. We are planning to go route 2 (pre-creating
and opening up perms on the dir) as a workaround, though I feel it's not a
good place to be if one is having to do that. There are a few different
directories that are needed to be manually created with the relaxed perms
to get around this (history, staging, tmp). Doing it outside of the regular
deployment process is not the best idea.

On Thu, Jun 20, 2013 at 12:54 AM, Chuan Liu <[EMAIL PROTECTED]> wrote:

>  Hi Prashant,****
>
> ** **
>
> We also hit this issue before.****
>
> 1) We want run a Hadoop cluster with permission disabled.****
>
> 2) With Job history server, yarn, hdfs daemons run under a special service
> user account, e.g. ‘hadoop’****
>
> 3) Users submit jobs to the cluster under their own account.****
>
> ** **
>
> For the above scenario, submitting jobs fails in Hadoop 2.0 while succeeds
> in Hadoop 1.0.****
>
> ** **
>
> In our investigation, the regression happened in jobclient and job history
> server, not on hdfs side.****
>
> The root cause is that jobclient will copy jar files to the staging area
> configed by “yarn.app.mapreduce.am.staging-dir”.****
>
> The client will also set the permission on the directory and jar files to
> some pre-configured value, i.e. JobSubmissionFilesJOB_DIR_PERMISSION and
> JobSubmissionFilesJOB_FILE_PERMISSION.****
>
> On HDFS side, even if ‘permissoin.enabled’ is set to false, changing
> permissions are not allowed.****
>
> (This is the same in both Hadoop v1 and v2.)****
>
> JobHistoryServer also plays a part in this as its staging directory
> happens to be at the same locations as “yarn.app.mapreduce am.staging-dir”.
> ****
>
> It will create directories recursively with permissions set to
> HISTORY_STAGING_DIR_PERMISSIONS.****
>
> JobHistoryServer runs under the special service user account while
> JobClient is under the user who submitting jobs.****
>
> This lead to a failure in setPermission() during job submission.****
>
> ** **
>
> There are multiple possible mitigations possible. Here are two examples.**
> **
>
> 1) config all users submitting jobs to supergroup.****
>
> 2) during setup, pre-create the staging directory and chown to correct
> user.****
>
> ** **
>
> In our case, we took approach 1) because the security check on HDFS was
> not very important for our scenarios (part of the reason why we can disable
> HDFS permission in the first place).****
>
> ** **
>
> Hope this can help you solve your problem!****
>
> ** **
>
> ** **
>
> -Chuan****
>
> ** **
>
> ** **
>
> *From:* Prashant Kommireddi [mailto:[EMAIL PROTECTED]]
> *Sent:* Wednesday, June 19, 2013 1:32 PM
> *To:* [EMAIL PROTECTED]
> *Subject:* Re: DFS Permissions on Hadoop 2.x****
>
> ** **
>
> How can we resolve the issue in the case I have mentioned? File a MR Jira
> that does not try to check permissions when dfs.permissions.enabled is set
> to false? ****
>
> ** **
>
> The explanation that Tsz Wo (Nicholas) pointed out in the JIRA makes sense
> w.r.t HDFS behavior (thanks for that). But I am still unsure how we can get
> around the fact that certain permissions are set on shared directories by a
> certain user that disallow any other users from using them. Or am I missing
> something entirely?****
>
> ** **
>
> On Wed, Jun 19, 2013 at 1:01 PM, Chris Nauroth <[EMAIL PROTECTED]>
> wrote:****
>
>  Just in case anyone is curious who didn't look at HDFS-4918, we
> established that this is actually expected behavior, and it's mentioned in
> the documentation.  However, I filed HDFS-4919 to make the information
> clearer in the documentation, since this caused some confusion.****
>
> ** **
>
> https://issues.apache.org/jira/browse/HDFS-4919****
>
> ** **
>
> Chris Nauroth****
>
> Hortonworks****
>
> http://hortonworks.com/****
>
> ** **
>
> ** **
>
> On Tue, Jun 18, 2013 at 10:42 PM, Prashant Kommireddi <[EMAIL PROTECTED]>
> wrote:****
>
>  Thanks guys, I will follow the discussion there.****
>
> ** **
>
> On Tue, Jun 18, 2013 at 10:10 PM, Azuryy Yu <[EMAIL PROTECTED]> wrote:***