Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
HDFS, mail # user - Re: DFS Permissions on Hadoop 2.x


+
Chris Nauroth 2013-06-18, 18:58
+
Leo Leung 2013-06-18, 19:14
+
Jean-Baptiste Onofré 2013-06-18, 19:07
+
Prashant Kommireddi 2013-06-19, 05:42
Copy link to this message
-
Re: DFS Permissions on Hadoop 2.x
Chris Nauroth 2013-06-19, 20:01
Just in case anyone is curious who didn't look at HDFS-4918, we established
that this is actually expected behavior, and it's mentioned in the
documentation.  However, I filed HDFS-4919 to make the information clearer
in the documentation, since this caused some confusion.

https://issues.apache.org/jira/browse/HDFS-4919

Chris Nauroth
Hortonworks
http://hortonworks.com/

On Tue, Jun 18, 2013 at 10:42 PM, Prashant Kommireddi
<[EMAIL PROTECTED]>wrote:

> Thanks guys, I will follow the discussion there.
>
>
> On Tue, Jun 18, 2013 at 10:10 PM, Azuryy Yu <[EMAIL PROTECTED]> wrote:
>
>> Yes, and I think this was lead by Snapshot.
>>
>> I've file a JIRA here:
>> https://issues.apache.org/jira/browse/HDFS-4918
>>
>>
>>
>> On Wed, Jun 19, 2013 at 11:40 AM, Harsh J <[EMAIL PROTECTED]> wrote:
>>
>>> This is a HDFS bug. Like all other methods that check for permissions
>>> being enabled, the client call of setPermission should check it as
>>> well. It does not do that currently and I believe it should be a NOP
>>> in such a case. Please do file a JIRA (and reference the ID here to
>>> close the loop)!
>>>
>>> On Wed, Jun 19, 2013 at 6:18 AM, Prashant Kommireddi
>>> <[EMAIL PROTECTED]> wrote:
>>> > Looks like the jobs fail only on the first attempt and pass thereafter.
>>> > Failure occurs while setting perms on "intermediate done directory".
>>> Here is
>>> > what I think is happening:
>>> >
>>> > 1. Intermediate done dir is (ideally) created as part of deployment
>>> (for eg,
>>> > /mapred/history/done_intermediate)
>>> >
>>> > 2. When a MR job is run, it creates a user dir within intermediate
>>> done dir
>>> > (/mapred/history/done_intermediate/username)
>>> >
>>> > 3. After this dir is created, the code tries to set permissions on
>>> this user
>>> > dir. In doing so, it checks for EXECUTE permissions on not just its
>>> parent
>>> > (/mapred/history/done_intermediate) but across all dirs to the top-most
>>> > level (/mapred). This fails as "/mapred" does not have execute
>>> permissions
>>> > for the "Other" users.
>>> >
>>> > 4. On successive job runs, since the user dir already exists
>>> > (/mapred/history/done_intermediate/username) it no longer tries to
>>> create
>>> > and set permissions again. And the job completes without any perm
>>> errors.
>>> >
>>> > This is the code within JobHistoryEventHandler that's doing it.
>>> >
>>> >  //Check for the existence of intermediate done dir.
>>> >     Path doneDirPath = null;
>>> >     try {
>>> >       doneDirPath = FileSystem.get(conf).makeQualified(new
>>> > Path(doneDirStr));
>>> >       doneDirFS = FileSystem.get(doneDirPath.toUri(), conf);
>>> >       // This directory will be in a common location, or this may be a
>>> > cluster
>>> >       // meant for a single user. Creating based on the conf. Should
>>> ideally
>>> > be
>>> >       // created by the JobHistoryServer or as part of deployment.
>>> >       if (!doneDirFS.exists(doneDirPath)) {
>>> >       if (JobHistoryUtils.shouldCreateNonUserDirectory(conf)) {
>>> >         LOG.info("Creating intermediate history logDir: ["
>>> >             + doneDirPath
>>> >             + "] + based on conf. Should ideally be created by the
>>> > JobHistoryServer: "
>>> >             + MRJobConfig.MR_AM_CREATE_JH_INTERMEDIATE_BASE_DIR);
>>> >           mkdir(
>>> >               doneDirFS,
>>> >               doneDirPath,
>>> >               new FsPermission(
>>> >             JobHistoryUtils.HISTORY_INTERMEDIATE_DONE_DIR_PERMISSIONS
>>> >                 .toShort()));
>>> >           // TODO Temporary toShort till new
>>> FsPermission(FsPermissions)
>>> >           // respects
>>> >         // sticky
>>> >       } else {
>>> >           String message = "Not creating intermediate history logDir:
>>> ["
>>> >                 + doneDirPath
>>> >                 + "] based on conf: "
>>> >                 + MRJobConfig.MR_AM_CREATE_JH_INTERMEDIATE_BASE_DIR
>>> >                 + ". Either set to true or pre-create this directory
>>> with" +
+
Prashant Kommireddi 2013-06-19, 20:31
+
Chuan Liu 2013-06-20, 07:54