|
|
-
startMiniDFSCluster and file permissions
lars hofhansl 2011-10-27, 21:53
I just noticed today that I could not run any test that starts a MiniDFSCluster.
The exception I got was this:
java.lang.NullPointerException
at org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes(MiniDFSCluster.java:422)
at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:280)
at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniDFSCluster(HBaseTestingUtility.java:350)
at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:519)
at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:475)
at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:462)
In the logs I had:
2011-10-27 14:17:48,238 WARN [main] datanode.DataNode(1540): Invalid directory in dfs.data.dir: Incorrect permission for /home/lars/dev/hbase-trunk/target/test-data/8f8d2437-1d9a-42fa-b7c3-c154d8e559f3/dfscluster_557b48bc-9c8e-4a47-b74e-4c0167710237/dfs/data/data1, expected: rwxr-xr-x, while actual: rwxrwxr-x
2011-10-27 14:17:48,260 WARN [main] datanode.DataNode(1540): Invalid directory in dfs.data.dir: Incorrect permission for /home/lars/dev/hbase-trunk/target/test-data/8f8d2437-1d9a-42fa-b7c3-c154d8e559f3/dfscluster_557b48bc-9c8e-4a47-b74e-4c0167710237/dfs/data/data2, expected: rwxr-xr-x, while actual: rwxrwxr-x
2011-10-27 14:17:48,261 ERROR [main] datanode.DataNode(1546): All directories in dfs.data.dir are invalid.
And indeed I see this in org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(...):
FsPermission dataDirPermission =
new FsPermission(conf.get(DATA_DIR_PERMISSION_KEY,
DEFAULT_DATA_DIR_PERMISSION));
for (String dir : dataDirs) {
try {
DiskChecker.checkDir(localFS, new Path(dir), dataDirPermission);
dirs.add(new File(dir));
} catch(IOException e) {
LOG.warn("Invalid directory in " + DATA_DIR_KEY + ": " +
e.getMessage());
}
}
(where DEFAULT_DATA_DIR_PERMISSION is 755)
The default umask on my machine is 0002, so that would seem to explain the discrepancy.
Changing my umask to 0022 fixed the problem!
I cannot be the only one seeing this. This is just a heads for anyone who runs into this, as I wasted over an hour on this.
I assume this is due to the switch to hadoop 0.20.205.
As I am fairly ignorant about Maven... Is there a way to set the default umask automatically for the test processes?
-- Lars
+
lars hofhansl 2011-10-27, 21:53
-
Re: startMiniDFSCluster and file permissions
Ted Yu 2011-10-27, 22:03
I think Apache Jenkins doesn't have this problem - otherwise we should have seen it by now. FYI: http://www.avajava.com/tutorials/lessons/how-do-i-set-the-default-file-and-directory-permissions.htmlOn Thu, Oct 27, 2011 at 2:53 PM, lars hofhansl <[EMAIL PROTECTED]> wrote: > I just noticed today that I could not run any test that starts a > MiniDFSCluster. > > The exception I got was this: > java.lang.NullPointerException > at > org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes(MiniDFSCluster.java:422) > at > org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:280) > at > org.apache.hadoop.hbase.HBaseTestingUtility.startMiniDFSCluster(HBaseTestingUtility.java:350) > at > org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:519) > at > org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:475) > at > org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:462) > > In the logs I had: > 2011-10-27 14:17:48,238 WARN [main] datanode.DataNode(1540): Invalid > directory in dfs.data.dir: Incorrect permission for > /home/lars/dev/hbase-trunk/target/test-data/8f8d2437-1d9a-42fa-b7c3-c154d8e559f3/dfscluster_557b48bc-9c8e-4a47-b74e-4c0167710237/dfs/data/data1, > expected: rwxr-xr-x, while actual: rwxrwxr-x > 2011-10-27 14:17:48,260 WARN [main] datanode.DataNode(1540): Invalid > directory in dfs.data.dir: Incorrect permission for > /home/lars/dev/hbase-trunk/target/test-data/8f8d2437-1d9a-42fa-b7c3-c154d8e559f3/dfscluster_557b48bc-9c8e-4a47-b74e-4c0167710237/dfs/data/data2, > expected: rwxr-xr-x, while actual: rwxrwxr-x > 2011-10-27 14:17:48,261 ERROR [main] datanode.DataNode(1546): All > directories in dfs.data.dir are invalid. > > > And indeed I see this in > org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(...): > > FsPermission dataDirPermission > new FsPermission(conf.get(DATA_DIR_PERMISSION_KEY, > DEFAULT_DATA_DIR_PERMISSION)); > for (String dir : dataDirs) { > try { > DiskChecker.checkDir(localFS, new Path(dir), dataDirPermission); > dirs.add(new File(dir)); > } catch(IOException e) { > LOG.warn("Invalid directory in " + DATA_DIR_KEY + ": " + > e.getMessage()); > } > } > > > (where DEFAULT_DATA_DIR_PERMISSION is 755) > > > The default umask on my machine is 0002, so that would seem to explain the > discrepancy. > > Changing my umask to 0022 fixed the problem! > I cannot be the only one seeing this. This is just a heads for anyone who > runs into this, as I wasted over an hour on this. > > I assume this is due to the switch to hadoop 0.20.205. > > As I am fairly ignorant about Maven... Is there a way to set the default > umask automatically for the test processes? > > -- Lars > >
+
Ted Yu 2011-10-27, 22:03
-
Re: startMiniDFSCluster and file permissions
lars hofhansl 2011-10-27, 22:20
I know how to change the default permissions (as mentioned in my email below)... The point is that I like my umask to be 0002. All users here are in their own group, so 0775 as default permission makes sense. Now I have to remember to set my umask to 0022 every time I want to run the tests, and since 0002 is the default on most Linux distributions I can't be the only one seeing this. On jenkins there must be a different default umask, or something else must be different. -- Lars ----- Original Message ----- From: Ted Yu <[EMAIL PROTECTED]> To: [EMAIL PROTECTED]; lars hofhansl <[EMAIL PROTECTED]> Cc: Sent: Thursday, October 27, 2011 3:03 PM Subject: Re: startMiniDFSCluster and file permissions I think Apache Jenkins doesn't have this problem - otherwise we should have seen it by now. FYI: http://www.avajava.com/tutorials/lessons/how-do-i-set-the-default-file-and-directory-permissions.htmlOn Thu, Oct 27, 2011 at 2:53 PM, lars hofhansl <[EMAIL PROTECTED]> wrote: > I just noticed today that I could not run any test that starts a > MiniDFSCluster. > > The exception I got was this: > java.lang.NullPointerException > at > org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes(MiniDFSCluster.java:422) > at > org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:280) > at > org.apache.hadoop.hbase.HBaseTestingUtility.startMiniDFSCluster(HBaseTestingUtility.java:350) > at > org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:519) > at > org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:475) > at > org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:462) > > In the logs I had: > 2011-10-27 14:17:48,238 WARN [main] datanode.DataNode(1540): Invalid > directory in dfs.data.dir: Incorrect permission for > /home/lars/dev/hbase-trunk/target/test-data/8f8d2437-1d9a-42fa-b7c3-c154d8e559f3/dfscluster_557b48bc-9c8e-4a47-b74e-4c0167710237/dfs/data/data1, > expected: rwxr-xr-x, while actual: rwxrwxr-x > 2011-10-27 14:17:48,260 WARN [main] datanode.DataNode(1540): Invalid > directory in dfs.data.dir: Incorrect permission for > /home/lars/dev/hbase-trunk/target/test-data/8f8d2437-1d9a-42fa-b7c3-c154d8e559f3/dfscluster_557b48bc-9c8e-4a47-b74e-4c0167710237/dfs/data/data2, > expected: rwxr-xr-x, while actual: rwxrwxr-x > 2011-10-27 14:17:48,261 ERROR [main] datanode.DataNode(1546): All > directories in dfs.data.dir are invalid. > > > And indeed I see this in > org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(...): > > FsPermission dataDirPermission > new FsPermission(conf.get(DATA_DIR_PERMISSION_KEY, > DEFAULT_DATA_DIR_PERMISSION)); > for (String dir : dataDirs) { > try { > DiskChecker.checkDir(localFS, new Path(dir), dataDirPermission); > dirs.add(new File(dir)); > } catch(IOException e) { > LOG.warn("Invalid directory in " + DATA_DIR_KEY + ": " + > e.getMessage()); > } > } > > > (where DEFAULT_DATA_DIR_PERMISSION is 755) > > > The default umask on my machine is 0002, so that would seem to explain the > discrepancy. > > Changing my umask to 0022 fixed the problem! > I cannot be the only one seeing this. This is just a heads for anyone who > runs into this, as I wasted over an hour on this. > > I assume this is due to the switch to hadoop 0.20.205. > > As I am fairly ignorant about Maven... Is there a way to set the default > umask automatically for the test processes? > > -- Lars > >
+
lars hofhansl 2011-10-27, 22:20
-
Re: startMiniDFSCluster and file permissions
Stack 2011-10-27, 22:41
Why don't we set DATA_DIR_PERMISSION_KEY to be permissive just before we spin up minidfscluster? St.Ack On Thu, Oct 27, 2011 at 3:03 PM, Ted Yu <[EMAIL PROTECTED]> wrote: > I think Apache Jenkins doesn't have this problem - otherwise we should have > seen it by now. > > FYI: > http://www.avajava.com/tutorials/lessons/how-do-i-set-the-default-file-and-directory-permissions.html> > On Thu, Oct 27, 2011 at 2:53 PM, lars hofhansl <[EMAIL PROTECTED]> wrote: > >> I just noticed today that I could not run any test that starts a >> MiniDFSCluster. >> >> The exception I got was this: >> java.lang.NullPointerException >> at >> org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes(MiniDFSCluster.java:422) >> at >> org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:280) >> at >> org.apache.hadoop.hbase.HBaseTestingUtility.startMiniDFSCluster(HBaseTestingUtility.java:350) >> at >> org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:519) >> at >> org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:475) >> at >> org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:462) >> >> In the logs I had: >> 2011-10-27 14:17:48,238 WARN [main] datanode.DataNode(1540): Invalid >> directory in dfs.data.dir: Incorrect permission for >> /home/lars/dev/hbase-trunk/target/test-data/8f8d2437-1d9a-42fa-b7c3-c154d8e559f3/dfscluster_557b48bc-9c8e-4a47-b74e-4c0167710237/dfs/data/data1, >> expected: rwxr-xr-x, while actual: rwxrwxr-x >> 2011-10-27 14:17:48,260 WARN [main] datanode.DataNode(1540): Invalid >> directory in dfs.data.dir: Incorrect permission for >> /home/lars/dev/hbase-trunk/target/test-data/8f8d2437-1d9a-42fa-b7c3-c154d8e559f3/dfscluster_557b48bc-9c8e-4a47-b74e-4c0167710237/dfs/data/data2, >> expected: rwxr-xr-x, while actual: rwxrwxr-x >> 2011-10-27 14:17:48,261 ERROR [main] datanode.DataNode(1546): All >> directories in dfs.data.dir are invalid. >> >> >> And indeed I see this in >> org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(...): >> >> FsPermission dataDirPermission >> new FsPermission(conf.get(DATA_DIR_PERMISSION_KEY, >> DEFAULT_DATA_DIR_PERMISSION)); >> for (String dir : dataDirs) { >> try { >> DiskChecker.checkDir(localFS, new Path(dir), dataDirPermission); >> dirs.add(new File(dir)); >> } catch(IOException e) { >> LOG.warn("Invalid directory in " + DATA_DIR_KEY + ": " + >> e.getMessage()); >> } >> } >> >> >> (where DEFAULT_DATA_DIR_PERMISSION is 755) >> >> >> The default umask on my machine is 0002, so that would seem to explain the >> discrepancy. >> >> Changing my umask to 0022 fixed the problem! >> I cannot be the only one seeing this. This is just a heads for anyone who >> runs into this, as I wasted over an hour on this. >> >> I assume this is due to the switch to hadoop 0.20.205. >> >> As I am fairly ignorant about Maven... Is there a way to set the default >> umask automatically for the test processes? >> >> -- Lars >> >> >
+
Stack 2011-10-27, 22:41
-
Re: startMiniDFSCluster and file permissions
lars hofhansl 2011-10-27, 23:16
Hmm... checkDir eventually calls checkPermission and that does an equals check on the expected and actual permissions. So we'd need to set DATA_DIR_PERMISSION_KEY to (777 XOR umask). Ugh. -- Lars ----- Original Message ----- From: Stack <[EMAIL PROTECTED]> To: [EMAIL PROTECTED] Cc: lars hofhansl <[EMAIL PROTECTED]> Sent: Thursday, October 27, 2011 3:41 PM Subject: Re: startMiniDFSCluster and file permissions Why don't we set DATA_DIR_PERMISSION_KEY to be permissive just before we spin up minidfscluster? St.Ack On Thu, Oct 27, 2011 at 3:03 PM, Ted Yu <[EMAIL PROTECTED]> wrote: > I think Apache Jenkins doesn't have this problem - otherwise we should have > seen it by now. > > FYI: > http://www.avajava.com/tutorials/lessons/how-do-i-set-the-default-file-and-directory-permissions.html> > On Thu, Oct 27, 2011 at 2:53 PM, lars hofhansl <[EMAIL PROTECTED]> wrote: > >> I just noticed today that I could not run any test that starts a >> MiniDFSCluster. >> >> The exception I got was this: >> java.lang.NullPointerException >> at >> org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes(MiniDFSCluster.java:422) >> at >> org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:280) >> at >> org.apache.hadoop.hbase.HBaseTestingUtility.startMiniDFSCluster(HBaseTestingUtility.java:350) >> at >> org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:519) >> at >> org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:475) >> at >> org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:462) >> >> In the logs I had: >> 2011-10-27 14:17:48,238 WARN [main] datanode.DataNode(1540): Invalid >> directory in dfs.data.dir: Incorrect permission for >> /home/lars/dev/hbase-trunk/target/test-data/8f8d2437-1d9a-42fa-b7c3-c154d8e559f3/dfscluster_557b48bc-9c8e-4a47-b74e-4c0167710237/dfs/data/data1, >> expected: rwxr-xr-x, while actual: rwxrwxr-x >> 2011-10-27 14:17:48,260 WARN [main] datanode.DataNode(1540): Invalid >> directory in dfs.data.dir: Incorrect permission for >> /home/lars/dev/hbase-trunk/target/test-data/8f8d2437-1d9a-42fa-b7c3-c154d8e559f3/dfscluster_557b48bc-9c8e-4a47-b74e-4c0167710237/dfs/data/data2, >> expected: rwxr-xr-x, while actual: rwxrwxr-x >> 2011-10-27 14:17:48,261 ERROR [main] datanode.DataNode(1546): All >> directories in dfs.data.dir are invalid. >> >> >> And indeed I see this in >> org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(...): >> >> FsPermission dataDirPermission >> new FsPermission(conf.get(DATA_DIR_PERMISSION_KEY, >> DEFAULT_DATA_DIR_PERMISSION)); >> for (String dir : dataDirs) { >> try { >> DiskChecker.checkDir(localFS, new Path(dir), dataDirPermission); >> dirs.add(new File(dir)); >> } catch(IOException e) { >> LOG.warn("Invalid directory in " + DATA_DIR_KEY + ": " + >> e.getMessage()); >> } >> } >> >> >> (where DEFAULT_DATA_DIR_PERMISSION is 755) >> >> >> The default umask on my machine is 0002, so that would seem to explain the >> discrepancy. >> >> Changing my umask to 0022 fixed the problem! >> I cannot be the only one seeing this. This is just a heads for anyone who >> runs into this, as I wasted over an hour on this. >> >> I assume this is due to the switch to hadoop 0.20.205. >> >> As I am fairly ignorant about Maven... Is there a way to set the default >> umask automatically for the test processes? >> >> -- Lars >> >> >
+
lars hofhansl 2011-10-27, 23:16
-
Re: startMiniDFSCluster and file permissions
Stack 2011-10-27, 23:18
On Thu, Oct 27, 2011 at 4:16 PM, lars hofhansl <[EMAIL PROTECTED]> wrote:
> Hmm... checkDir eventually calls checkPermission and that does an equals check on the expected and actual permissions.
>
> So we'd need to set DATA_DIR_PERMISSION_KEY to (777 XOR umask). Ugh.
>
We should do it?
St.Ack
+
Stack 2011-10-27, 23:18
-
Re: startMiniDFSCluster and file permissions
Gary Helmling 2011-10-27, 23:20
This is fixed by HDFS-1560, though unfortunately it's not in 0.20.205.0. I've just been running with umask set to 022. On Thu, Oct 27, 2011 at 4:16 PM, lars hofhansl <[EMAIL PROTECTED]> wrote: > Hmm... checkDir eventually calls checkPermission and that does an equals check on the expected and actual permissions. > > So we'd need to set DATA_DIR_PERMISSION_KEY to (777 XOR umask). Ugh. > > > -- Lars > > > ----- Original Message ----- > From: Stack <[EMAIL PROTECTED]> > To: [EMAIL PROTECTED] > Cc: lars hofhansl <[EMAIL PROTECTED]> > Sent: Thursday, October 27, 2011 3:41 PM > Subject: Re: startMiniDFSCluster and file permissions > > Why don't we set DATA_DIR_PERMISSION_KEY to be permissive just before > we spin up minidfscluster? > St.Ack > > > > On Thu, Oct 27, 2011 at 3:03 PM, Ted Yu <[EMAIL PROTECTED]> wrote: >> I think Apache Jenkins doesn't have this problem - otherwise we should have >> seen it by now. >> >> FYI: >> http://www.avajava.com/tutorials/lessons/how-do-i-set-the-default-file-and-directory-permissions.html>> >> On Thu, Oct 27, 2011 at 2:53 PM, lars hofhansl <[EMAIL PROTECTED]> wrote: >> >>> I just noticed today that I could not run any test that starts a >>> MiniDFSCluster. >>> >>> The exception I got was this: >>> java.lang.NullPointerException >>> at >>> org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes(MiniDFSCluster.java:422) >>> at >>> org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:280) >>> at >>> org.apache.hadoop.hbase.HBaseTestingUtility.startMiniDFSCluster(HBaseTestingUtility.java:350) >>> at >>> org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:519) >>> at >>> org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:475) >>> at >>> org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:462) >>> >>> In the logs I had: >>> 2011-10-27 14:17:48,238 WARN [main] datanode.DataNode(1540): Invalid >>> directory in dfs.data.dir: Incorrect permission for >>> /home/lars/dev/hbase-trunk/target/test-data/8f8d2437-1d9a-42fa-b7c3-c154d8e559f3/dfscluster_557b48bc-9c8e-4a47-b74e-4c0167710237/dfs/data/data1, >>> expected: rwxr-xr-x, while actual: rwxrwxr-x >>> 2011-10-27 14:17:48,260 WARN [main] datanode.DataNode(1540): Invalid >>> directory in dfs.data.dir: Incorrect permission for >>> /home/lars/dev/hbase-trunk/target/test-data/8f8d2437-1d9a-42fa-b7c3-c154d8e559f3/dfscluster_557b48bc-9c8e-4a47-b74e-4c0167710237/dfs/data/data2, >>> expected: rwxr-xr-x, while actual: rwxrwxr-x >>> 2011-10-27 14:17:48,261 ERROR [main] datanode.DataNode(1546): All >>> directories in dfs.data.dir are invalid. >>> >>> >>> And indeed I see this in >>> org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(...): >>> >>> FsPermission dataDirPermission >>> new FsPermission(conf.get(DATA_DIR_PERMISSION_KEY, >>> DEFAULT_DATA_DIR_PERMISSION)); >>> for (String dir : dataDirs) { >>> try { >>> DiskChecker.checkDir(localFS, new Path(dir), dataDirPermission); >>> dirs.add(new File(dir)); >>> } catch(IOException e) { >>> LOG.warn("Invalid directory in " + DATA_DIR_KEY + ": " + >>> e.getMessage()); >>> } >>> } >>> >>> >>> (where DEFAULT_DATA_DIR_PERMISSION is 755) >>> >>> >>> The default umask on my machine is 0002, so that would seem to explain the >>> discrepancy. >>> >>> Changing my umask to 0022 fixed the problem! >>> I cannot be the only one seeing this. This is just a heads for anyone who >>> runs into this, as I wasted over an hour on this. >>> >>> I assume this is due to the switch to hadoop 0.20.205. >>> >>> As I am fairly ignorant about Maven... Is there a way to set the default >>> umask automatically for the test processes? >>> >>> -- Lars >>> >>> >> > >
+
Gary Helmling 2011-10-27, 23:20
-
Re: startMiniDFSCluster and file permissions
lars hofhansl 2011-10-27, 23:34
Wouldn't that still break in the same way only that you'd need to set umask to 077 in order to avoid it? ----- Original Message ----- From: Gary Helmling <[EMAIL PROTECTED]> To: [EMAIL PROTECTED]; lars hofhansl <[EMAIL PROTECTED]> Cc: Sent: Thursday, October 27, 2011 4:20 PM Subject: Re: startMiniDFSCluster and file permissions This is fixed by HDFS-1560, though unfortunately it's not in 0.20.205.0. I've just been running with umask set to 022. On Thu, Oct 27, 2011 at 4:16 PM, lars hofhansl <[EMAIL PROTECTED]> wrote: > Hmm... checkDir eventually calls checkPermission and that does an equals check on the expected and actual permissions. > > So we'd need to set DATA_DIR_PERMISSION_KEY to (777 XOR umask). Ugh. > > > -- Lars > > > ----- Original Message ----- > From: Stack <[EMAIL PROTECTED]> > To: [EMAIL PROTECTED] > Cc: lars hofhansl <[EMAIL PROTECTED]> > Sent: Thursday, October 27, 2011 3:41 PM > Subject: Re: startMiniDFSCluster and file permissions > > Why don't we set DATA_DIR_PERMISSION_KEY to be permissive just before > we spin up minidfscluster? > St.Ack > > > > On Thu, Oct 27, 2011 at 3:03 PM, Ted Yu <[EMAIL PROTECTED]> wrote: >> I think Apache Jenkins doesn't have this problem - otherwise we should have >> seen it by now. >> >> FYI: >> http://www.avajava.com/tutorials/lessons/how-do-i-set-the-default-file-and-directory-permissions.html>> >> On Thu, Oct 27, 2011 at 2:53 PM, lars hofhansl <[EMAIL PROTECTED]> wrote: >> >>> I just noticed today that I could not run any test that starts a >>> MiniDFSCluster. >>> >>> The exception I got was this: >>> java.lang.NullPointerException >>> at >>> org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes(MiniDFSCluster.java:422) >>> at >>> org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:280) >>> at >>> org.apache.hadoop.hbase.HBaseTestingUtility.startMiniDFSCluster(HBaseTestingUtility.java:350) >>> at >>> org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:519) >>> at >>> org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:475) >>> at >>> org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:462) >>> >>> In the logs I had: >>> 2011-10-27 14:17:48,238 WARN [main] datanode.DataNode(1540): Invalid >>> directory in dfs.data.dir: Incorrect permission for >>> /home/lars/dev/hbase-trunk/target/test-data/8f8d2437-1d9a-42fa-b7c3-c154d8e559f3/dfscluster_557b48bc-9c8e-4a47-b74e-4c0167710237/dfs/data/data1, >>> expected: rwxr-xr-x, while actual: rwxrwxr-x >>> 2011-10-27 14:17:48,260 WARN [main] datanode.DataNode(1540): Invalid >>> directory in dfs.data.dir: Incorrect permission for >>> /home/lars/dev/hbase-trunk/target/test-data/8f8d2437-1d9a-42fa-b7c3-c154d8e559f3/dfscluster_557b48bc-9c8e-4a47-b74e-4c0167710237/dfs/data/data2, >>> expected: rwxr-xr-x, while actual: rwxrwxr-x >>> 2011-10-27 14:17:48,261 ERROR [main] datanode.DataNode(1546): All >>> directories in dfs.data.dir are invalid. >>> >>> >>> And indeed I see this in >>> org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(...): >>> >>> FsPermission dataDirPermission >>> new FsPermission(conf.get(DATA_DIR_PERMISSION_KEY, >>> DEFAULT_DATA_DIR_PERMISSION)); >>> for (String dir : dataDirs) { >>> try { >>> DiskChecker.checkDir(localFS, new Path(dir), dataDirPermission); >>> dirs.add(new File(dir)); >>> } catch(IOException e) { >>> LOG.warn("Invalid directory in " + DATA_DIR_KEY + ": " + >>> e.getMessage()); >>> } >>> } >>> >>> >>> (where DEFAULT_DATA_DIR_PERMISSION is 755) >>> >>> >>> The default umask on my machine is 0002, so that would seem to explain the >>> discrepancy. >>> >>> Changing my umask to 0022 fixed the problem! >>> I cannot be the only one seeing this. This is just a heads for anyone who >>> runs into this, as I wasted over an hour on this.
+
lars hofhansl 2011-10-27, 23:34
-
Re: startMiniDFSCluster and file permissions
Gary Helmling 2011-10-27, 23:40
No, the HDFS-1560 change actually sets the permissions on the directories if they don't match the expected. The current behavior is just to bail out if they don't match, so you have to orchestrate your umask so the default permissions wind up right. On Thu, Oct 27, 2011 at 4:34 PM, lars hofhansl <[EMAIL PROTECTED]> wrote: > Wouldn't that still break in the same way only that you'd need to set umask to 077 in order to avoid it? > > > > ----- Original Message ----- > From: Gary Helmling <[EMAIL PROTECTED]> > To: [EMAIL PROTECTED]; lars hofhansl <[EMAIL PROTECTED]> > Cc: > Sent: Thursday, October 27, 2011 4:20 PM > Subject: Re: startMiniDFSCluster and file permissions > > This is fixed by HDFS-1560, though unfortunately it's not in > 0.20.205.0. I've just been running with umask set to 022. > > > On Thu, Oct 27, 2011 at 4:16 PM, lars hofhansl <[EMAIL PROTECTED]> wrote: >> Hmm... checkDir eventually calls checkPermission and that does an equals check on the expected and actual permissions. >> >> So we'd need to set DATA_DIR_PERMISSION_KEY to (777 XOR umask). Ugh. >> >> >> -- Lars >> >> >> ----- Original Message ----- >> From: Stack <[EMAIL PROTECTED]> >> To: [EMAIL PROTECTED] >> Cc: lars hofhansl <[EMAIL PROTECTED]> >> Sent: Thursday, October 27, 2011 3:41 PM >> Subject: Re: startMiniDFSCluster and file permissions >> >> Why don't we set DATA_DIR_PERMISSION_KEY to be permissive just before >> we spin up minidfscluster? >> St.Ack >> >> >> >> On Thu, Oct 27, 2011 at 3:03 PM, Ted Yu <[EMAIL PROTECTED]> wrote: >>> I think Apache Jenkins doesn't have this problem - otherwise we should have >>> seen it by now. >>> >>> FYI: >>> http://www.avajava.com/tutorials/lessons/how-do-i-set-the-default-file-and-directory-permissions.html>>> >>> On Thu, Oct 27, 2011 at 2:53 PM, lars hofhansl <[EMAIL PROTECTED]> wrote: >>> >>>> I just noticed today that I could not run any test that starts a >>>> MiniDFSCluster. >>>> >>>> The exception I got was this: >>>> java.lang.NullPointerException >>>> at >>>> org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes(MiniDFSCluster.java:422) >>>> at >>>> org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:280) >>>> at >>>> org.apache.hadoop.hbase.HBaseTestingUtility.startMiniDFSCluster(HBaseTestingUtility.java:350) >>>> at >>>> org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:519) >>>> at >>>> org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:475) >>>> at >>>> org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:462) >>>> >>>> In the logs I had: >>>> 2011-10-27 14:17:48,238 WARN [main] datanode.DataNode(1540): Invalid >>>> directory in dfs.data.dir: Incorrect permission for >>>> /home/lars/dev/hbase-trunk/target/test-data/8f8d2437-1d9a-42fa-b7c3-c154d8e559f3/dfscluster_557b48bc-9c8e-4a47-b74e-4c0167710237/dfs/data/data1, >>>> expected: rwxr-xr-x, while actual: rwxrwxr-x >>>> 2011-10-27 14:17:48,260 WARN [main] datanode.DataNode(1540): Invalid >>>> directory in dfs.data.dir: Incorrect permission for >>>> /home/lars/dev/hbase-trunk/target/test-data/8f8d2437-1d9a-42fa-b7c3-c154d8e559f3/dfscluster_557b48bc-9c8e-4a47-b74e-4c0167710237/dfs/data/data2, >>>> expected: rwxr-xr-x, while actual: rwxrwxr-x >>>> 2011-10-27 14:17:48,261 ERROR [main] datanode.DataNode(1546): All >>>> directories in dfs.data.dir are invalid. >>>> >>>> >>>> And indeed I see this in >>>> org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(...): >>>> >>>> FsPermission dataDirPermission >>>> new FsPermission(conf.get(DATA_DIR_PERMISSION_KEY, >>>> DEFAULT_DATA_DIR_PERMISSION)); >>>> for (String dir : dataDirs) { >>>> try { >>>> DiskChecker.checkDir(localFS, new Path(dir), dataDirPermission); >>>> dirs.add(new File(dir)); >>>> } catch(IOException e) { >>>> LOG.warn("Invalid directory in " + DATA_DIR_KEY + ": " +
+
Gary Helmling 2011-10-27, 23:40
-
Re: startMiniDFSCluster and file permissions
Andrew Purtell 2011-10-27, 23:38
IMHO, HDFS-1560 should go into 0.20.205.1. There are a couple other patches we've found useful and have kept floating on 205: https://github.com/trendmicro/hadoop-common/commits/work/ - HDFS-611 improves performance of a HBase cluster HDFS-1205 names the threads introduced by HDFS-611 for use in troubleshooting - HADOOP-5124 / HDFS-1257 are a more direct way to deal with delete-related performance issues on a HBase cluster, by addressing the underlying delete work distribution issue in HDFS 0.20.x. Actually without this patch I've seen DNs in small clusters become extremely unbalanced when HBase is under heavy write load (triggering a high rate of compactions). The commit in the above referenced tree is a port of the combination of the two jiras. HDFS-611 is still good to have. - HADOOP-6522 fixes decoding of codepoint zero in UTF8. Without this change TestUTF8 often fails for me. Best regards, - Andy Problems worthy of attack prove their worth by hitting back. - Piet Hein (via Tom White) >________________________________ >From: Gary Helmling <[EMAIL PROTECTED]> >To: [EMAIL PROTECTED]; lars hofhansl <[EMAIL PROTECTED]> >Sent: Thursday, October 27, 2011 4:20 PM >Subject: Re: startMiniDFSCluster and file permissions > >This is fixed by HDFS-1560, though unfortunately it's not in >0.20.205.0. I've just been running with umask set to 022. > > >On Thu, Oct 27, 2011 at 4:16 PM, lars hofhansl <[EMAIL PROTECTED]> wrote: >> Hmm... checkDir eventually calls checkPermission and that does an equals check on the expected and actual permissions. >> >> So we'd need to set DATA_DIR_PERMISSION_KEY to (777 XOR umask). Ugh. >> >> >> -- Lars >> >> >> --->> From: Stack <[EMAIL PROTECTED]> >> To: [EMAIL PROTECTED] >> Cc: lars hofhansl <[EMAIL PROTECTED]> >> Sent: Thursday, October 27, 2011 3:41 PM >> Subject: Re: startMiniDFSCluster and file permissions >> >> Why don't we set DATA_DIR_PERMISSION_KEY to be permissive just before >> we spin up minidfscluster? >> St.Ack >> >> >> >> On Thu, Oct 27, 2011 at 3:03 PM, Ted Yu <[EMAIL PROTECTED]> wrote: >>> I think Apache Jenkins doesn't have this problem - otherwise we should have >>> seen it by now. >>> >>> FYI: >>> http://www.avajava.com/tutorials/lessons/how-do-i-set-the-default-file-and-directory-permissions.html>>> >>> On Thu, Oct 27, 2011 at 2:53 PM, lars hofhansl <[EMAIL PROTECTED]> wrote: >>> >>>> I just noticed today that I could not run any test that starts a >>>> MiniDFSCluster. >>>> >>>> The exception I got was this: >>>> java.lang.NullPointerException >>>> at >>>> org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes(MiniDFSCluster.java:422) >>>> at >>>> org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:280) >>>> at >>>> org.apache.hadoop.hbase.HBaseTestingUtility.startMiniDFSCluster(HBaseTestingUtility.java:350) >>>> at >>>> org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:519) >>>> at >>>> org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:475) >>>> at >>>> org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:462) >>>> >>>> In the logs I had: >>>> 2011-10-27 14:17:48,238 WARN [main] datanode.DataNode(1540): Invalid >>>> directory in dfs.data.dir: Incorrect permission for >>>> /home/lars/dev/hbase-trunk/target/test-data/8f8d2437-1d9a-42fa-b7c3-c154d8e559f3/dfscluster_557b48bc-9c8e-4a47-b74e-4c0167710237/dfs/data/data1, >>>> expected: rwxr-xr-x, while actual: rwxrwxr-x >>>> 2011-10-27 14:17:48,260 WARN [main] datanode.DataNode(1540): Invalid >>>> directory in dfs.data.dir: Incorrect permission for >>>> /home/lars/dev/hbase-trunk/target/test-data/8f8d2437-1d9a-42fa-b7c3-c154d8e559f3/dfscluster_557b48bc-9c8e-4a47-b74e-4c0167710237/dfs/data/data2, >>>> expected: rwxr-xr-x, while actual: rwxrwxr-x >>>> 2011-10-27 14:17:48,261 ERROR [main] datanode.DataNode(1546): All >>>> directories in dfs.data.dir are invalid.
+
Andrew Purtell 2011-10-27, 23:38
-
Re: startMiniDFSCluster and file permissions
lars hofhansl 2011-10-28, 01:09
Thanks Andrew. This is very useful information for us. ----- Original Message ----- From: Andrew Purtell <[EMAIL PROTECTED]> To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>; lars hofhansl <[EMAIL PROTECTED]>; Matt Foley <[EMAIL PROTECTED]> Cc: Sent: Thursday, October 27, 2011 4:38 PM Subject: Re: startMiniDFSCluster and file permissions IMHO, HDFS-1560 should go into 0.20.205.1. There are a couple other patches we've found useful and have kept floating on 205: https://github.com/trendmicro/hadoop-common/commits/work/ - HDFS-611 improves performance of a HBase cluster HDFS-1205 names the threads introduced by HDFS-611 for use in troubleshooting - HADOOP-5124 / HDFS-1257 are a more direct way to deal with delete-related performance issues on a HBase cluster, by addressing the underlying delete work distribution issue in HDFS 0.20.x. Actually without this patch I've seen DNs in small clusters become extremely unbalanced when HBase is under heavy write load (triggering a high rate of compactions). The commit in the above referenced tree is a port of the combination of the two jiras. HDFS-611 is still good to have. - HADOOP-6522 fixes decoding of codepoint zero in UTF8. Without this change TestUTF8 often fails for me. Best regards, - Andy Problems worthy of attack prove their worth by hitting back. - Piet Hein (via Tom White) >________________________________ >From: Gary Helmling <[EMAIL PROTECTED]> >To: [EMAIL PROTECTED]; lars hofhansl <[EMAIL PROTECTED]> >Sent: Thursday, October 27, 2011 4:20 PM >Subject: Re: startMiniDFSCluster and file permissions > >This is fixed by HDFS-1560, though unfortunately it's not in >0.20.205.0. I've just been running with umask set to 022. > > >On Thu, Oct 27, 2011 at 4:16 PM, lars hofhansl <[EMAIL PROTECTED]> wrote: >> Hmm... checkDir eventually calls checkPermission and that does an equals check on the expected and actual permissions. >> >> So we'd need to set DATA_DIR_PERMISSION_KEY to (777 XOR umask). Ugh. >> >> >> -- Lars >> >> >> ----- Original Message ----- >> From: Stack <[EMAIL PROTECTED]> >> To: [EMAIL PROTECTED] >> Cc: lars hofhansl <[EMAIL PROTECTED]> >> Sent: Thursday, October 27, 2011 3:41 PM >> Subject: Re: startMiniDFSCluster and file permissions >> >> Why don't we set DATA_DIR_PERMISSION_KEY to be permissive just before >> we spin up minidfscluster? >> St.Ack >> >> >> >> On Thu, Oct 27, 2011 at 3:03 PM, Ted Yu <[EMAIL PROTECTED]> wrote: >>> I think Apache Jenkins doesn't have this problem - otherwise we should have >>> seen it by now. >>> >>> FYI: >>> http://www.avajava.com/tutorials/lessons/how-do-i-set-the-default-file-and-directory-permissions.html>>> >>> On Thu, Oct 27, 2011 at 2:53 PM, lars hofhansl <[EMAIL PROTECTED]> wrote: >>> >>>> I just noticed today that I could not run any test that starts a >>>> MiniDFSCluster. >>>> >>>> The exception I got was this: >>>> java.lang.NullPointerException >>>> at >>>> org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes(MiniDFSCluster.java:422) >>>> at >>>> org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:280) >>>> at >>>> org.apache.hadoop.hbase.HBaseTestingUtility.startMiniDFSCluster(HBaseTestingUtility.java:350) >>>> at >>>> org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:519) >>>> at >>>> org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:475) >>>> at >>>> org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:462) >>>> >>>> In the logs I had: >>>> 2011-10-27 14:17:48,238 WARN [main] datanode.DataNode(1540): Invalid >>>> directory in dfs.data.dir: Incorrect permission for >>>> /home/lars/dev/hbase-trunk/target/test-data/8f8d2437-1d9a-42fa-b7c3-c154d8e559f3/dfscluster_557b48bc-9c8e-4a47-b74e-4c0167710237/dfs/data/data1, >>>> expected: rwxr-xr-x, while actual: rwxrwxr-x >>>> 2011-10-27 14:17:48,260 WARN [main] datanode.DataNode(1540): Invalid
+
lars hofhansl 2011-10-28, 01:09
|
|
