Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
HDFS >> mail # user >> Re: DFS Permissions on Hadoop 2.x


+
Chris Nauroth 2013-06-18, 18:58
+
Leo Leung 2013-06-18, 19:14
+
Jean-Baptiste Onofré 2013-06-18, 19:07
+
Prashant Kommireddi 2013-06-19, 05:42
Copy link to this message
-
Re: DFS Permissions on Hadoop 2.x
Just in case anyone is curious who didn't look at HDFS-4918, we established
that this is actually expected behavior, and it's mentioned in the
documentation.  However, I filed HDFS-4919 to make the information clearer
in the documentation, since this caused some confusion.

https://issues.apache.org/jira/browse/HDFS-4919

Chris Nauroth
Hortonworks
http://hortonworks.com/

On Tue, Jun 18, 2013 at 10:42 PM, Prashant Kommireddi
<[EMAIL PROTECTED]>wrote:

> Thanks guys, I will follow the discussion there.
>
>
> On Tue, Jun 18, 2013 at 10:10 PM, Azuryy Yu <[EMAIL PROTECTED]> wrote:
>
>> Yes, and I think this was lead by Snapshot.
>>
>> I've file a JIRA here:
>> https://issues.apache.org/jira/browse/HDFS-4918
>>
>>
>>
>> On Wed, Jun 19, 2013 at 11:40 AM, Harsh J <[EMAIL PROTECTED]> wrote:
>>
>>> This is a HDFS bug. Like all other methods that check for permissions
>>> being enabled, the client call of setPermission should check it as
>>> well. It does not do that currently and I believe it should be a NOP
>>> in such a case. Please do file a JIRA (and reference the ID here to
>>> close the loop)!
>>>
>>> On Wed, Jun 19, 2013 at 6:18 AM, Prashant Kommireddi
>>> <[EMAIL PROTECTED]> wrote:
>>> > Looks like the jobs fail only on the first attempt and pass thereafter.
>>> > Failure occurs while setting perms on "intermediate done directory".
>>> Here is
>>> > what I think is happening:
>>> >
>>> > 1. Intermediate done dir is (ideally) created as part of deployment
>>> (for eg,
>>> > /mapred/history/done_intermediate)
>>> >
>>> > 2. When a MR job is run, it creates a user dir within intermediate
>>> done dir
>>> > (/mapred/history/done_intermediate/username)
>>> >
>>> > 3. After this dir is created, the code tries to set permissions on
>>> this user
>>> > dir. In doing so, it checks for EXECUTE permissions on not just its
>>> parent
>>> > (/mapred/history/done_intermediate) but across all dirs to the top-most
>>> > level (/mapred). This fails as "/mapred" does not have execute
>>> permissions
>>> > for the "Other" users.
>>> >
>>> > 4. On successive job runs, since the user dir already exists
>>> > (/mapred/history/done_intermediate/username) it no longer tries to
>>> create
>>> > and set permissions again. And the job completes without any perm
>>> errors.
>>> >
>>> > This is the code within JobHistoryEventHandler that's doing it.
>>> >
>>> >  //Check for the existence of intermediate done dir.
>>> >     Path doneDirPath = null;
>>> >     try {
>>> >       doneDirPath = FileSystem.get(conf).makeQualified(new
>>> > Path(doneDirStr));
>>> >       doneDirFS = FileSystem.get(doneDirPath.toUri(), conf);
>>> >       // This directory will be in a common location, or this may be a
>>> > cluster
>>> >       // meant for a single user. Creating based on the conf. Should
>>> ideally
>>> > be
>>> >       // created by the JobHistoryServer or as part of deployment.
>>> >       if (!doneDirFS.exists(doneDirPath)) {
>>> >       if (JobHistoryUtils.shouldCreateNonUserDirectory(conf)) {
>>> >         LOG.info("Creating intermediate history logDir: ["
>>> >             + doneDirPath
>>> >             + "] + based on conf. Should ideally be created by the
>>> > JobHistoryServer: "
>>> >             + MRJobConfig.MR_AM_CREATE_JH_INTERMEDIATE_BASE_DIR);
>>> >           mkdir(
>>> >               doneDirFS,
>>> >               doneDirPath,
>>> >               new FsPermission(
>>> >             JobHistoryUtils.HISTORY_INTERMEDIATE_DONE_DIR_PERMISSIONS
>>> >                 .toShort()));
>>> >           // TODO Temporary toShort till new
>>> FsPermission(FsPermissions)
>>> >           // respects
>>> >         // sticky
>>> >       } else {
>>> >           String message = "Not creating intermediate history logDir:
>>> ["
>>> >                 + doneDirPath
>>> >                 + "] based on conf: "
>>> >                 + MRJobConfig.MR_AM_CREATE_JH_INTERMEDIATE_BASE_DIR
>>> >                 + ". Either set to true or pre-create this directory
>>> with" +
+
Prashant Kommireddi 2013-06-19, 20:31
+
Chuan Liu 2013-06-20, 07:54
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB