Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # user >> Re: DFS Permissions on Hadoop 2.x


Copy link to this message
-
Re: DFS Permissions on Hadoop 2.x
Yes, and I think this was lead by Snapshot.

I've file a JIRA here:
https://issues.apache.org/jira/browse/HDFS-4918

On Wed, Jun 19, 2013 at 11:40 AM, Harsh J <[EMAIL PROTECTED]> wrote:

> This is a HDFS bug. Like all other methods that check for permissions
> being enabled, the client call of setPermission should check it as
> well. It does not do that currently and I believe it should be a NOP
> in such a case. Please do file a JIRA (and reference the ID here to
> close the loop)!
>
> On Wed, Jun 19, 2013 at 6:18 AM, Prashant Kommireddi
> <[EMAIL PROTECTED]> wrote:
> > Looks like the jobs fail only on the first attempt and pass thereafter.
> > Failure occurs while setting perms on "intermediate done directory".
> Here is
> > what I think is happening:
> >
> > 1. Intermediate done dir is (ideally) created as part of deployment (for
> eg,
> > /mapred/history/done_intermediate)
> >
> > 2. When a MR job is run, it creates a user dir within intermediate done
> dir
> > (/mapred/history/done_intermediate/username)
> >
> > 3. After this dir is created, the code tries to set permissions on this
> user
> > dir. In doing so, it checks for EXECUTE permissions on not just its
> parent
> > (/mapred/history/done_intermediate) but across all dirs to the top-most
> > level (/mapred). This fails as "/mapred" does not have execute
> permissions
> > for the "Other" users.
> >
> > 4. On successive job runs, since the user dir already exists
> > (/mapred/history/done_intermediate/username) it no longer tries to create
> > and set permissions again. And the job completes without any perm errors.
> >
> > This is the code within JobHistoryEventHandler that's doing it.
> >
> >  //Check for the existence of intermediate done dir.
> >     Path doneDirPath = null;
> >     try {
> >       doneDirPath = FileSystem.get(conf).makeQualified(new
> > Path(doneDirStr));
> >       doneDirFS = FileSystem.get(doneDirPath.toUri(), conf);
> >       // This directory will be in a common location, or this may be a
> > cluster
> >       // meant for a single user. Creating based on the conf. Should
> ideally
> > be
> >       // created by the JobHistoryServer or as part of deployment.
> >       if (!doneDirFS.exists(doneDirPath)) {
> >       if (JobHistoryUtils.shouldCreateNonUserDirectory(conf)) {
> >         LOG.info("Creating intermediate history logDir: ["
> >             + doneDirPath
> >             + "] + based on conf. Should ideally be created by the
> > JobHistoryServer: "
> >             + MRJobConfig.MR_AM_CREATE_JH_INTERMEDIATE_BASE_DIR);
> >           mkdir(
> >               doneDirFS,
> >               doneDirPath,
> >               new FsPermission(
> >             JobHistoryUtils.HISTORY_INTERMEDIATE_DONE_DIR_PERMISSIONS
> >                 .toShort()));
> >           // TODO Temporary toShort till new FsPermission(FsPermissions)
> >           // respects
> >         // sticky
> >       } else {
> >           String message = "Not creating intermediate history logDir: ["
> >                 + doneDirPath
> >                 + "] based on conf: "
> >                 + MRJobConfig.MR_AM_CREATE_JH_INTERMEDIATE_BASE_DIR
> >                 + ". Either set to true or pre-create this directory
> with" +
> >                 " appropriate permissions";
> >         LOG.error(message);
> >         throw new YarnException(message);
> >       }
> >       }
> >     } catch (IOException e) {
> >       LOG.error("Failed checking for the existance of history
> intermediate "
> > +
> >                       "done directory: [" + doneDirPath + "]");
> >       throw new YarnException(e);
> >     }
> >
> >
> > In any case, this does not appear to be the right behavior as it does not
> > respect "dfs.permissions.enabled" (set to false) at any point. Sounds
> like a
> > bug?
> >
> >
> > Thanks, Prashant
> >
> >
> >
> >
> >
> >
> > On Tue, Jun 18, 2013 at 3:24 PM, Prashant Kommireddi <
> [EMAIL PROTECTED]>
> > wrote:
> >>
> >> Hi Chris,
> >>
> >> This is while running a MR job. Please note the job is able to write