Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Hadoop >> mail # user >> Re: DFS Permissions on Hadoop 2.x


Copy link to this message
-
Re: DFS Permissions on Hadoop 2.x
Yes, and I think this was lead by Snapshot.

I've file a JIRA here:
https://issues.apache.org/jira/browse/HDFS-4918

On Wed, Jun 19, 2013 at 11:40 AM, Harsh J <[EMAIL PROTECTED]> wrote:

> This is a HDFS bug. Like all other methods that check for permissions
> being enabled, the client call of setPermission should check it as
> well. It does not do that currently and I believe it should be a NOP
> in such a case. Please do file a JIRA (and reference the ID here to
> close the loop)!
>
> On Wed, Jun 19, 2013 at 6:18 AM, Prashant Kommireddi
> <[EMAIL PROTECTED]> wrote:
> > Looks like the jobs fail only on the first attempt and pass thereafter.
> > Failure occurs while setting perms on "intermediate done directory".
> Here is
> > what I think is happening:
> >
> > 1. Intermediate done dir is (ideally) created as part of deployment (for
> eg,
> > /mapred/history/done_intermediate)
> >
> > 2. When a MR job is run, it creates a user dir within intermediate done
> dir
> > (/mapred/history/done_intermediate/username)
> >
> > 3. After this dir is created, the code tries to set permissions on this
> user
> > dir. In doing so, it checks for EXECUTE permissions on not just its
> parent
> > (/mapred/history/done_intermediate) but across all dirs to the top-most
> > level (/mapred). This fails as "/mapred" does not have execute
> permissions
> > for the "Other" users.
> >
> > 4. On successive job runs, since the user dir already exists
> > (/mapred/history/done_intermediate/username) it no longer tries to create
> > and set permissions again. And the job completes without any perm errors.
> >
> > This is the code within JobHistoryEventHandler that's doing it.
> >
> >  //Check for the existence of intermediate done dir.
> >     Path doneDirPath = null;
> >     try {
> >       doneDirPath = FileSystem.get(conf).makeQualified(new
> > Path(doneDirStr));
> >       doneDirFS = FileSystem.get(doneDirPath.toUri(), conf);
> >       // This directory will be in a common location, or this may be a
> > cluster
> >       // meant for a single user. Creating based on the conf. Should
> ideally
> > be
> >       // created by the JobHistoryServer or as part of deployment.
> >       if (!doneDirFS.exists(doneDirPath)) {
> >       if (JobHistoryUtils.shouldCreateNonUserDirectory(conf)) {
> >         LOG.info("Creating intermediate history logDir: ["
> >             + doneDirPath
> >             + "] + based on conf. Should ideally be created by the
> > JobHistoryServer: "
> >             + MRJobConfig.MR_AM_CREATE_JH_INTERMEDIATE_BASE_DIR);
> >           mkdir(
> >               doneDirFS,
> >               doneDirPath,
> >               new FsPermission(
> >             JobHistoryUtils.HISTORY_INTERMEDIATE_DONE_DIR_PERMISSIONS
> >                 .toShort()));
> >           // TODO Temporary toShort till new FsPermission(FsPermissions)
> >           // respects
> >         // sticky
> >       } else {
> >           String message = "Not creating intermediate history logDir: ["
> >                 + doneDirPath
> >                 + "] based on conf: "
> >                 + MRJobConfig.MR_AM_CREATE_JH_INTERMEDIATE_BASE_DIR
> >                 + ". Either set to true or pre-create this directory
> with" +
> >                 " appropriate permissions";
> >         LOG.error(message);
> >         throw new YarnException(message);
> >       }
> >       }
> >     } catch (IOException e) {
> >       LOG.error("Failed checking for the existance of history
> intermediate "
> > +
> >                       "done directory: [" + doneDirPath + "]");
> >       throw new YarnException(e);
> >     }
> >
> >
> > In any case, this does not appear to be the right behavior as it does not
> > respect "dfs.permissions.enabled" (set to false) at any point. Sounds
> like a
> > bug?
> >
> >
> > Thanks, Prashant
> >
> >
> >
> >
> >
> >
> > On Tue, Jun 18, 2013 at 3:24 PM, Prashant Kommireddi <
> [EMAIL PROTECTED]>
> > wrote:
> >>
> >> Hi Chris,
> >>
> >> This is while running a MR job. Please note the job is able to write
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB