Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
MapReduce, mail # user - DFS Permissions on Hadoop 2.x


+
Prashant Kommireddi 2013-06-18, 17:54
+
Prashant Kommireddi 2013-06-18, 19:04
+
Chris Nauroth 2013-06-18, 20:28
+
Prashant Kommireddi 2013-06-18, 22:24
+
Prashant Kommireddi 2013-06-19, 00:48
Copy link to this message
-
Re: DFS Permissions on Hadoop 2.x
Harsh J 2013-06-19, 03:40
This is a HDFS bug. Like all other methods that check for permissions
being enabled, the client call of setPermission should check it as
well. It does not do that currently and I believe it should be a NOP
in such a case. Please do file a JIRA (and reference the ID here to
close the loop)!

On Wed, Jun 19, 2013 at 6:18 AM, Prashant Kommireddi
<[EMAIL PROTECTED]> wrote:
> Looks like the jobs fail only on the first attempt and pass thereafter.
> Failure occurs while setting perms on "intermediate done directory". Here is
> what I think is happening:
>
> 1. Intermediate done dir is (ideally) created as part of deployment (for eg,
> /mapred/history/done_intermediate)
>
> 2. When a MR job is run, it creates a user dir within intermediate done dir
> (/mapred/history/done_intermediate/username)
>
> 3. After this dir is created, the code tries to set permissions on this user
> dir. In doing so, it checks for EXECUTE permissions on not just its parent
> (/mapred/history/done_intermediate) but across all dirs to the top-most
> level (/mapred). This fails as "/mapred" does not have execute permissions
> for the "Other" users.
>
> 4. On successive job runs, since the user dir already exists
> (/mapred/history/done_intermediate/username) it no longer tries to create
> and set permissions again. And the job completes without any perm errors.
>
> This is the code within JobHistoryEventHandler that's doing it.
>
>  //Check for the existence of intermediate done dir.
>     Path doneDirPath = null;
>     try {
>       doneDirPath = FileSystem.get(conf).makeQualified(new
> Path(doneDirStr));
>       doneDirFS = FileSystem.get(doneDirPath.toUri(), conf);
>       // This directory will be in a common location, or this may be a
> cluster
>       // meant for a single user. Creating based on the conf. Should ideally
> be
>       // created by the JobHistoryServer or as part of deployment.
>       if (!doneDirFS.exists(doneDirPath)) {
>       if (JobHistoryUtils.shouldCreateNonUserDirectory(conf)) {
>         LOG.info("Creating intermediate history logDir: ["
>             + doneDirPath
>             + "] + based on conf. Should ideally be created by the
> JobHistoryServer: "
>             + MRJobConfig.MR_AM_CREATE_JH_INTERMEDIATE_BASE_DIR);
>           mkdir(
>               doneDirFS,
>               doneDirPath,
>               new FsPermission(
>             JobHistoryUtils.HISTORY_INTERMEDIATE_DONE_DIR_PERMISSIONS
>                 .toShort()));
>           // TODO Temporary toShort till new FsPermission(FsPermissions)
>           // respects
>         // sticky
>       } else {
>           String message = "Not creating intermediate history logDir: ["
>                 + doneDirPath
>                 + "] based on conf: "
>                 + MRJobConfig.MR_AM_CREATE_JH_INTERMEDIATE_BASE_DIR
>                 + ". Either set to true or pre-create this directory with" +
>                 " appropriate permissions";
>         LOG.error(message);
>         throw new YarnException(message);
>       }
>       }
>     } catch (IOException e) {
>       LOG.error("Failed checking for the existance of history intermediate "
> +
>       "done directory: [" + doneDirPath + "]");
>       throw new YarnException(e);
>     }
>
>
> In any case, this does not appear to be the right behavior as it does not
> respect "dfs.permissions.enabled" (set to false) at any point. Sounds like a
> bug?
>
>
> Thanks, Prashant
>
>
>
>
>
>
> On Tue, Jun 18, 2013 at 3:24 PM, Prashant Kommireddi <[EMAIL PROTECTED]>
> wrote:
>>
>> Hi Chris,
>>
>> This is while running a MR job. Please note the job is able to write files
>> to "/mapred" directory and fails on EXECUTE permissions. On digging in some
>> more, it looks like the failure occurs after writing to
>> "/mapred/history/done_intermediate".
>>
>> Here is a more detailed stacktrace.
>>
>> INFO: Job end notification started for jobID : job_1371593763906_0001
>> Jun 18, 2013 3:20:20 PM
>> org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler

Harsh J
+
Prashant Kommireddi 2013-06-20, 21:16