Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce >> mail # user >> DFS Permissions on Hadoop 2.x


Copy link to this message
-
Re: DFS Permissions on Hadoop 2.x
Looks like the jobs fail only on the first attempt and pass thereafter.
Failure occurs while setting perms on "intermediate done directory". Here
is what I think is happening:

1. Intermediate done dir is (ideally) created as part of deployment (for
eg, /mapred/history/done_intermediate)

2. When a MR job is run, it creates a user dir within intermediate done dir
(/mapred/history/done_intermediate/username)

3. After this dir is created, the code tries to set permissions on this
user dir. In doing so, it checks for EXECUTE permissions on not just its
parent (/mapred/history/done_intermediate) but across all dirs to the
top-most level (/mapred). This fails as "/mapred" does not have execute
permissions for the "Other" users.

4. On successive job runs, since the user dir already exists
(/mapred/history/done_intermediate/username) it no longer tries to create
and set permissions again. And the job completes without any perm errors.

This is the code within JobHistoryEventHandler that's doing it.

 //Check for the existence of intermediate done dir.
    Path doneDirPath = null;
    try {
      doneDirPath = FileSystem.get(conf).makeQualified(new Path(doneDirStr));
      doneDirFS = FileSystem.get(doneDirPath.toUri(), conf);
      // This directory will be in a common location, or this may be a cluster
      // meant for a single user. Creating based on the conf. Should ideally be
      // created by the JobHistoryServer or as part of deployment.
      if (!doneDirFS.exists(doneDirPath)) {
      if (JobHistoryUtils.shouldCreateNonUserDirectory(conf)) {
        LOG.info("Creating intermediate history logDir: ["
            + doneDirPath
            + "] + based on conf. Should ideally be created by the
JobHistoryServer: "
            + MRJobConfig.MR_AM_CREATE_JH_INTERMEDIATE_BASE_DIR);
          mkdir(
              doneDirFS,
              doneDirPath,
              new FsPermission(
            JobHistoryUtils.HISTORY_INTERMEDIATE_DONE_DIR_PERMISSIONS
                .toShort()));
          // TODO Temporary toShort till new FsPermission(FsPermissions)
          // respects
        // sticky
      } else {
          String message = "Not creating intermediate history logDir: ["
                + doneDirPath
                + "] based on conf: "
                + MRJobConfig.MR_AM_CREATE_JH_INTERMEDIATE_BASE_DIR
                + ". Either set to true or pre-create this directory with" +
                " appropriate permissions";
        LOG.error(message);
        throw new YarnException(message);
      }
      }
    } catch (IOException e) {
      LOG.error("Failed checking for the existance of history intermediate " +
       "done directory: [" + doneDirPath + "]");
      throw new YarnException(e);
    }
In any case, this does not appear to be the right behavior as it does
not respect "dfs.permissions.enabled" (set to false) at any point.
Sounds like a bug?
Thanks, Prashant
On Tue, Jun 18, 2013 at 3:24 PM, Prashant Kommireddi <[EMAIL PROTECTED]>wrote:

> Hi Chris,
>
> This is while running a MR job. Please note the job is able to write files
> to "/mapred" directory and fails on EXECUTE permissions. On digging in some
> more, it looks like the failure occurs after writing to
> "/mapred/history/done_intermediate".
>
> Here is a more detailed stacktrace.
>
> INFO: Job end notification started for jobID : job_1371593763906_0001
> Jun 18, 2013 3:20:20 PM org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler closeEventWriter
> INFO: Unable to write out JobSummaryInfo to [hdfs://test-local-EMPTYSPEC/mapred/history/done_intermediate/smehta/job_1371593763906_0001.summary_tmp]
> org.apache.hadoop.security.AccessControlException: Permission denied: user=smehta, access=EXECUTE, inode="/mapred":pkommireddi:supergroup:drwxrwx---
> at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:205)
> at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkTraverse(FSPermissionChecker.java:161)
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB