Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce >> mail # user >> DFS Permissions on Hadoop 2.x


Copy link to this message
-
Re: DFS Permissions on Hadoop 2.x
Looks like the jobs fail only on the first attempt and pass thereafter.
Failure occurs while setting perms on "intermediate done directory". Here
is what I think is happening:

1. Intermediate done dir is (ideally) created as part of deployment (for
eg, /mapred/history/done_intermediate)

2. When a MR job is run, it creates a user dir within intermediate done dir
(/mapred/history/done_intermediate/username)

3. After this dir is created, the code tries to set permissions on this
user dir. In doing so, it checks for EXECUTE permissions on not just its
parent (/mapred/history/done_intermediate) but across all dirs to the
top-most level (/mapred). This fails as "/mapred" does not have execute
permissions for the "Other" users.

4. On successive job runs, since the user dir already exists
(/mapred/history/done_intermediate/username) it no longer tries to create
and set permissions again. And the job completes without any perm errors.

This is the code within JobHistoryEventHandler that's doing it.

 //Check for the existence of intermediate done dir.
    Path doneDirPath = null;
    try {
      doneDirPath = FileSystem.get(conf).makeQualified(new Path(doneDirStr));
      doneDirFS = FileSystem.get(doneDirPath.toUri(), conf);
      // This directory will be in a common location, or this may be a cluster
      // meant for a single user. Creating based on the conf. Should ideally be
      // created by the JobHistoryServer or as part of deployment.
      if (!doneDirFS.exists(doneDirPath)) {
      if (JobHistoryUtils.shouldCreateNonUserDirectory(conf)) {
        LOG.info("Creating intermediate history logDir: ["
            + doneDirPath
            + "] + based on conf. Should ideally be created by the
JobHistoryServer: "
            + MRJobConfig.MR_AM_CREATE_JH_INTERMEDIATE_BASE_DIR);
          mkdir(
              doneDirFS,
              doneDirPath,
              new FsPermission(
            JobHistoryUtils.HISTORY_INTERMEDIATE_DONE_DIR_PERMISSIONS
                .toShort()));
          // TODO Temporary toShort till new FsPermission(FsPermissions)
          // respects
        // sticky
      } else {
          String message = "Not creating intermediate history logDir: ["
                + doneDirPath
                + "] based on conf: "
                + MRJobConfig.MR_AM_CREATE_JH_INTERMEDIATE_BASE_DIR
                + ". Either set to true or pre-create this directory with" +
                " appropriate permissions";
        LOG.error(message);
        throw new YarnException(message);
      }
      }
    } catch (IOException e) {
      LOG.error("Failed checking for the existance of history intermediate " +
       "done directory: [" + doneDirPath + "]");
      throw new YarnException(e);
    }
In any case, this does not appear to be the right behavior as it does
not respect "dfs.permissions.enabled" (set to false) at any point.
Sounds like a bug?
Thanks, Prashant
On Tue, Jun 18, 2013 at 3:24 PM, Prashant Kommireddi <[EMAIL PROTECTED]>wrote:

> Hi Chris,
>
> This is while running a MR job. Please note the job is able to write files
> to "/mapred" directory and fails on EXECUTE permissions. On digging in some
> more, it looks like the failure occurs after writing to
> "/mapred/history/done_intermediate".
>
> Here is a more detailed stacktrace.
>
> INFO: Job end notification started for jobID : job_1371593763906_0001
> Jun 18, 2013 3:20:20 PM org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler closeEventWriter
> INFO: Unable to write out JobSummaryInfo to [hdfs://test-local-EMPTYSPEC/mapred/history/done_intermediate/smehta/job_1371593763906_0001.summary_tmp]
> org.apache.hadoop.security.AccessControlException: Permission denied: user=smehta, access=EXECUTE, inode="/mapred":pkommireddi:supergroup:drwxrwx---
> at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:205)
> at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkTraverse(FSPermissionChecker.java:161)