Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce >> mail # user >> DFS Permissions on Hadoop 2.x


Copy link to this message
-
Re: DFS Permissions on Hadoop 2.x
This is a HDFS bug. Like all other methods that check for permissions
being enabled, the client call of setPermission should check it as
well. It does not do that currently and I believe it should be a NOP
in such a case. Please do file a JIRA (and reference the ID here to
close the loop)!

On Wed, Jun 19, 2013 at 6:18 AM, Prashant Kommireddi
<[EMAIL PROTECTED]> wrote:
> Looks like the jobs fail only on the first attempt and pass thereafter.
> Failure occurs while setting perms on "intermediate done directory". Here is
> what I think is happening:
>
> 1. Intermediate done dir is (ideally) created as part of deployment (for eg,
> /mapred/history/done_intermediate)
>
> 2. When a MR job is run, it creates a user dir within intermediate done dir
> (/mapred/history/done_intermediate/username)
>
> 3. After this dir is created, the code tries to set permissions on this user
> dir. In doing so, it checks for EXECUTE permissions on not just its parent
> (/mapred/history/done_intermediate) but across all dirs to the top-most
> level (/mapred). This fails as "/mapred" does not have execute permissions
> for the "Other" users.
>
> 4. On successive job runs, since the user dir already exists
> (/mapred/history/done_intermediate/username) it no longer tries to create
> and set permissions again. And the job completes without any perm errors.
>
> This is the code within JobHistoryEventHandler that's doing it.
>
>  //Check for the existence of intermediate done dir.
>     Path doneDirPath = null;
>     try {
>       doneDirPath = FileSystem.get(conf).makeQualified(new
> Path(doneDirStr));
>       doneDirFS = FileSystem.get(doneDirPath.toUri(), conf);
>       // This directory will be in a common location, or this may be a
> cluster
>       // meant for a single user. Creating based on the conf. Should ideally
> be
>       // created by the JobHistoryServer or as part of deployment.
>       if (!doneDirFS.exists(doneDirPath)) {
>       if (JobHistoryUtils.shouldCreateNonUserDirectory(conf)) {
>         LOG.info("Creating intermediate history logDir: ["
>             + doneDirPath
>             + "] + based on conf. Should ideally be created by the
> JobHistoryServer: "
>             + MRJobConfig.MR_AM_CREATE_JH_INTERMEDIATE_BASE_DIR);
>           mkdir(
>               doneDirFS,
>               doneDirPath,
>               new FsPermission(
>             JobHistoryUtils.HISTORY_INTERMEDIATE_DONE_DIR_PERMISSIONS
>                 .toShort()));
>           // TODO Temporary toShort till new FsPermission(FsPermissions)
>           // respects
>         // sticky
>       } else {
>           String message = "Not creating intermediate history logDir: ["
>                 + doneDirPath
>                 + "] based on conf: "
>                 + MRJobConfig.MR_AM_CREATE_JH_INTERMEDIATE_BASE_DIR
>                 + ". Either set to true or pre-create this directory with" +
>                 " appropriate permissions";
>         LOG.error(message);
>         throw new YarnException(message);
>       }
>       }
>     } catch (IOException e) {
>       LOG.error("Failed checking for the existance of history intermediate "
> +
>       "done directory: [" + doneDirPath + "]");
>       throw new YarnException(e);
>     }
>
>
> In any case, this does not appear to be the right behavior as it does not
> respect "dfs.permissions.enabled" (set to false) at any point. Sounds like a
> bug?
>
>
> Thanks, Prashant
>
>
>
>
>
>
> On Tue, Jun 18, 2013 at 3:24 PM, Prashant Kommireddi <[EMAIL PROTECTED]>
> wrote:
>>
>> Hi Chris,
>>
>> This is while running a MR job. Please note the job is able to write files
>> to "/mapred" directory and fails on EXECUTE permissions. On digging in some
>> more, it looks like the failure occurs after writing to
>> "/mapred/history/done_intermediate".
>>
>> Here is a more detailed stacktrace.
>>
>> INFO: Job end notification started for jobID : job_1371593763906_0001
>> Jun 18, 2013 3:20:20 PM
>> org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler

Harsh J
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB