n keywal
2012-09-13, 14:58
Jean-Daniel Cryans
2012-09-13, 16:30
n keywal
2012-09-13, 16:42
Jean-Daniel Cryans
2012-09-13, 16:51
Stack
2012-09-13, 17:28
Andrew Purtell
2012-09-13, 17:44
Andrew Purtell
2012-09-13, 17:46
Jesse Yates
2012-09-13, 17:57
Jean-Daniel Cryans
2012-09-13, 18:01
Andrew Purtell
2012-09-13, 18:31
n keywal
2012-09-13, 19:17
Andrew Purtell
2012-09-13, 20:20
n keywal
2012-09-13, 20:34
Andrew Purtell
2012-09-13, 21:07
n keywal
2012-09-14, 12:48
-
read short circuitn keywal 2012-09-13, 14:58
Hi,
Is there any known counter indication for read short circuit?
I had a look at JD's presentation from January. It was tested vs. HBase as
a whole; and still recommended for nearly all cases.
We could make it at least the default when reading HLog files (even if it
seems it's a 'per client" config, there is may be something to look at as
well)?
Cheers,
Nicolas
Is there any known counter indication for read short circuit?
I had a look at JD's presentation from January. It was tested vs. HBase as
a whole; and still recommended for nearly all cases.
We could make it at least the default when reading HLog files (even if it
seems it's a 'per client" config, there is may be something to look at as
well)?
Cheers,
Nicolas
-
Re: read short circuitJean-Daniel Cryans 2012-09-13, 16:30
On Thu, Sep 13, 2012 at 7:58 AM, n keywal <[EMAIL PROTECTED]> wrote:
> Hi,
>
> Is there any known counter indication for read short circuit?
> I had a look at JD's presentation from January. It was tested vs. HBase as
> a whole; and still recommended for nearly all cases.
Pretty much.
>
> We could make it at least the default when reading HLog files (even if it
> seems it's a 'per client" config, there is may be something to look at as
> well)?
You have to add 2 configs in hdfs-site.xml, which we can't do from HBase.
J-D
> Hi,
>
> Is there any known counter indication for read short circuit?
> I had a look at JD's presentation from January. It was tested vs. HBase as
> a whole; and still recommended for nearly all cases.
Pretty much.
>
> We could make it at least the default when reading HLog files (even if it
> seems it's a 'per client" config, there is may be something to look at as
> well)?
You have to add 2 configs in hdfs-site.xml, which we can't do from HBase.
J-D
-
Re: read short circuitn keywal 2012-09-13, 16:42
It's all in the "pretty much" :-)
Should we make it the default (i.e. recommending it in the doc, emitting a
warning if it's not set) for all installation ?
Or would it make sense to separate the store and wal cases, meaning that we
need a change in hdfs to make this possible?
On Thu, Sep 13, 2012 at 6:30 PM, Jean-Daniel Cryans <[EMAIL PROTECTED]>wrote:
> On Thu, Sep 13, 2012 at 7:58 AM, n keywal <[EMAIL PROTECTED]> wrote:
> > Hi,
> >
> > Is there any known counter indication for read short circuit?
> > I had a look at JD's presentation from January. It was tested vs. HBase
> as
> > a whole; and still recommended for nearly all cases.
>
> Pretty much.
>
> >
> > We could make it at least the default when reading HLog files (even if it
> > seems it's a 'per client" config, there is may be something to look at as
> > well)?
>
> You have to add 2 configs in hdfs-site.xml, which we can't do from HBase.
>
> J-D
>
Should we make it the default (i.e. recommending it in the doc, emitting a
warning if it's not set) for all installation ?
Or would it make sense to separate the store and wal cases, meaning that we
need a change in hdfs to make this possible?
On Thu, Sep 13, 2012 at 6:30 PM, Jean-Daniel Cryans <[EMAIL PROTECTED]>wrote:
> On Thu, Sep 13, 2012 at 7:58 AM, n keywal <[EMAIL PROTECTED]> wrote:
> > Hi,
> >
> > Is there any known counter indication for read short circuit?
> > I had a look at JD's presentation from January. It was tested vs. HBase
> as
> > a whole; and still recommended for nearly all cases.
>
> Pretty much.
>
> >
> > We could make it at least the default when reading HLog files (even if it
> > seems it's a 'per client" config, there is may be something to look at as
> > well)?
>
> You have to add 2 configs in hdfs-site.xml, which we can't do from HBase.
>
> J-D
>
-
Re: read short circuitJean-Daniel Cryans 2012-09-13, 16:51
On Thu, Sep 13, 2012 at 9:42 AM, n keywal <[EMAIL PROTECTED]> wrote:
> It's all in the "pretty much" :-)
> Should we make it the default (i.e. recommending it in the doc, emitting a
> warning if it's not set) for all installation ?
I'd recommend it in the doc as an important config to consider.
One issue with the current implementation is that you can only have 1
user that short circuit reads, all the others who have the config
enabled will get a fat Exception. At SU we don't have special config
files for every MR job so unless you disable it job by job then you'll
get those exceptions everywhere. We live with it but it did take
people by surprise when we rolled it out :)
> Or would it make sense to separate the store and wal cases, meaning that we
> need a change in hdfs to make this possible?
Don't we only have one FS instance per region server? If so, that's a
moot point.
J-D
> It's all in the "pretty much" :-)
> Should we make it the default (i.e. recommending it in the doc, emitting a
> warning if it's not set) for all installation ?
I'd recommend it in the doc as an important config to consider.
One issue with the current implementation is that you can only have 1
user that short circuit reads, all the others who have the config
enabled will get a fat Exception. At SU we don't have special config
files for every MR job so unless you disable it job by job then you'll
get those exceptions everywhere. We live with it but it did take
people by surprise when we rolled it out :)
> Or would it make sense to separate the store and wal cases, meaning that we
> need a change in hdfs to make this possible?
Don't we only have one FS instance per region server? If so, that's a
moot point.
J-D
-
Re: read short circuitStack 2012-09-13, 17:28
On Thu, Sep 13, 2012 at 9:51 AM, Jean-Daniel Cryans <[EMAIL PROTECTED]> wrote:
> On Thu, Sep 13, 2012 at 9:42 AM, n keywal <[EMAIL PROTECTED]> wrote:
>> It's all in the "pretty much" :-)
>> Should we make it the default (i.e. recommending it in the doc, emitting a
>> warning if it's not set) for all installation ?
>
> I'd recommend it in the doc as an important config to consider.
>
> One issue with the current implementation is that you can only have 1
> user that short circuit reads, all the others who have the config
> enabled will get a fat Exception. At SU we don't have special config
> files for every MR job so unless you disable it job by job then you'll
> get those exceptions everywhere. We live with it but it did take
> people by surprise when we rolled it out :)
>
>> Or would it make sense to separate the store and wal cases, meaning that we
>> need a change in hdfs to make this possible?
>
> Don't we only have one FS instance per region server? If so, that's a
> moot point.
>
Write a short paragraph and I'll make an HDFS configuration sections
like this HBase configurations section on manual and stick it in
there: http://hbase.apache.org/book.html#perf.configurations
St.Ack
> On Thu, Sep 13, 2012 at 9:42 AM, n keywal <[EMAIL PROTECTED]> wrote:
>> It's all in the "pretty much" :-)
>> Should we make it the default (i.e. recommending it in the doc, emitting a
>> warning if it's not set) for all installation ?
>
> I'd recommend it in the doc as an important config to consider.
>
> One issue with the current implementation is that you can only have 1
> user that short circuit reads, all the others who have the config
> enabled will get a fat Exception. At SU we don't have special config
> files for every MR job so unless you disable it job by job then you'll
> get those exceptions everywhere. We live with it but it did take
> people by surprise when we rolled it out :)
>
>> Or would it make sense to separate the store and wal cases, meaning that we
>> need a change in hdfs to make this possible?
>
> Don't we only have one FS instance per region server? If so, that's a
> moot point.
>
Write a short paragraph and I'll make an HDFS configuration sections
like this HBase configurations section on manual and stick it in
there: http://hbase.apache.org/book.html#perf.configurations
St.Ack
-
Re: read short circuitAndrew Purtell 2012-09-13, 17:44
Making it the default seems reasonable.
For <= 0.94, the security profile has a separate hbase-site.xml file
under test resources where there it could be defaulted as off. Reason
is the local block sharing requires some shared group membership
between HBase and HDFS service accounts that might be objectionable in
a secure setting. So we should have test coverage with it off.
On Thu, Sep 13, 2012 at 9:42 AM, n keywal <[EMAIL PROTECTED]> wrote:
> It's all in the "pretty much" :-)
> Should we make it the default (i.e. recommending it in the doc, emitting a
> warning if it's not set) for all installation ?
> Or would it make sense to separate the store and wal cases, meaning that we
> need a change in hdfs to make this possible?
>
>
> On Thu, Sep 13, 2012 at 6:30 PM, Jean-Daniel Cryans <[EMAIL PROTECTED]>wrote:
>
>> On Thu, Sep 13, 2012 at 7:58 AM, n keywal <[EMAIL PROTECTED]> wrote:
>> > Hi,
>> >
>> > Is there any known counter indication for read short circuit?
>> > I had a look at JD's presentation from January. It was tested vs. HBase
>> as
>> > a whole; and still recommended for nearly all cases.
>>
>> Pretty much.
>>
>> >
>> > We could make it at least the default when reading HLog files (even if it
>> > seems it's a 'per client" config, there is may be something to look at as
>> > well)?
>>
>> You have to add 2 configs in hdfs-site.xml, which we can't do from HBase.
>>
>> J-D
>>
--
Best regards,
- Andy
Problems worthy of attack prove their worth by hitting back. - Piet
Hein (via Tom White)
For <= 0.94, the security profile has a separate hbase-site.xml file
under test resources where there it could be defaulted as off. Reason
is the local block sharing requires some shared group membership
between HBase and HDFS service accounts that might be objectionable in
a secure setting. So we should have test coverage with it off.
On Thu, Sep 13, 2012 at 9:42 AM, n keywal <[EMAIL PROTECTED]> wrote:
> It's all in the "pretty much" :-)
> Should we make it the default (i.e. recommending it in the doc, emitting a
> warning if it's not set) for all installation ?
> Or would it make sense to separate the store and wal cases, meaning that we
> need a change in hdfs to make this possible?
>
>
> On Thu, Sep 13, 2012 at 6:30 PM, Jean-Daniel Cryans <[EMAIL PROTECTED]>wrote:
>
>> On Thu, Sep 13, 2012 at 7:58 AM, n keywal <[EMAIL PROTECTED]> wrote:
>> > Hi,
>> >
>> > Is there any known counter indication for read short circuit?
>> > I had a look at JD's presentation from January. It was tested vs. HBase
>> as
>> > a whole; and still recommended for nearly all cases.
>>
>> Pretty much.
>>
>> >
>> > We could make it at least the default when reading HLog files (even if it
>> > seems it's a 'per client" config, there is may be something to look at as
>> > well)?
>>
>> You have to add 2 configs in hdfs-site.xml, which we can't do from HBase.
>>
>> J-D
>>
--
Best regards,
- Andy
Problems worthy of attack prove their worth by hitting back. - Piet
Hein (via Tom White)
-
Re: read short circuitAndrew Purtell 2012-09-13, 17:46
On Thu, Sep 13, 2012 at 9:51 AM, Jean-Daniel Cryans <[EMAIL PROTECTED]> wrote:
> One issue with the current implementation is that you can only have 1
> user that short circuit reads, all the others who have the config
> enabled will get a fat Exception. At SU we don't have special config
> files for every MR job so unless you disable it job by job then you'll
> get those exceptions everywhere. We live with it but it did take
> people by surprise when we rolled it out :)
Eli just fixed that in HDFS-3907. It's a three line change that
substitutes a list of strings for a single string, plus a unit test.
Should be a low risk change to get into HDFSes everywhere.
Best regards,
- Andy
Problems worthy of attack prove their worth by hitting back. - Piet
Hein (via Tom White)
> One issue with the current implementation is that you can only have 1
> user that short circuit reads, all the others who have the config
> enabled will get a fat Exception. At SU we don't have special config
> files for every MR job so unless you disable it job by job then you'll
> get those exceptions everywhere. We live with it but it did take
> people by surprise when we rolled it out :)
Eli just fixed that in HDFS-3907. It's a three line change that
substitutes a list of strings for a single string, plus a unit test.
Should be a low risk change to get into HDFSes everywhere.
Best regards,
- Andy
Problems worthy of attack prove their worth by hitting back. - Piet
Hein (via Tom White)
-
Re: read short circuitJesse Yates 2012-09-13, 17:57
There was a jira a while back (can't remember the number) that I think came
from the Facebook fellas to have multiple hbase-site.xmls, with a naming
scheme such that you can do layering of configurations.
What ever happened to that?
-------------------
Jesse Yates
@jesse_yates
jyates.github.com
On Thu, Sep 13, 2012 at 10:44 AM, Andrew Purtell <[EMAIL PROTECTED]>wrote:
> Making it the default seems reasonable.
>
> For <= 0.94, the security profile has a separate hbase-site.xml file
> under test resources where there it could be defaulted as off. Reason
> is the local block sharing requires some shared group membership
> between HBase and HDFS service accounts that might be objectionable in
> a secure setting. So we should have test coverage with it off.
>
> On Thu, Sep 13, 2012 at 9:42 AM, n keywal <[EMAIL PROTECTED]> wrote:
> > It's all in the "pretty much" :-)
> > Should we make it the default (i.e. recommending it in the doc, emitting
> a
> > warning if it's not set) for all installation ?
> > Or would it make sense to separate the store and wal cases, meaning that
> we
> > need a change in hdfs to make this possible?
> >
> >
> > On Thu, Sep 13, 2012 at 6:30 PM, Jean-Daniel Cryans <[EMAIL PROTECTED]
> >wrote:
> >
> >> On Thu, Sep 13, 2012 at 7:58 AM, n keywal <[EMAIL PROTECTED]> wrote:
> >> > Hi,
> >> >
> >> > Is there any known counter indication for read short circuit?
> >> > I had a look at JD's presentation from January. It was tested vs.
> HBase
> >> as
> >> > a whole; and still recommended for nearly all cases.
> >>
> >> Pretty much.
> >>
> >> >
> >> > We could make it at least the default when reading HLog files (even
> if it
> >> > seems it's a 'per client" config, there is may be something to look
> at as
> >> > well)?
> >>
> >> You have to add 2 configs in hdfs-site.xml, which we can't do from
> HBase.
> >>
> >> J-D
> >>
>
>
>
> --
> Best regards,
>
> - Andy
>
> Problems worthy of attack prove their worth by hitting back. - Piet
> Hein (via Tom White)
>
from the Facebook fellas to have multiple hbase-site.xmls, with a naming
scheme such that you can do layering of configurations.
What ever happened to that?
-------------------
Jesse Yates
@jesse_yates
jyates.github.com
On Thu, Sep 13, 2012 at 10:44 AM, Andrew Purtell <[EMAIL PROTECTED]>wrote:
> Making it the default seems reasonable.
>
> For <= 0.94, the security profile has a separate hbase-site.xml file
> under test resources where there it could be defaulted as off. Reason
> is the local block sharing requires some shared group membership
> between HBase and HDFS service accounts that might be objectionable in
> a secure setting. So we should have test coverage with it off.
>
> On Thu, Sep 13, 2012 at 9:42 AM, n keywal <[EMAIL PROTECTED]> wrote:
> > It's all in the "pretty much" :-)
> > Should we make it the default (i.e. recommending it in the doc, emitting
> a
> > warning if it's not set) for all installation ?
> > Or would it make sense to separate the store and wal cases, meaning that
> we
> > need a change in hdfs to make this possible?
> >
> >
> > On Thu, Sep 13, 2012 at 6:30 PM, Jean-Daniel Cryans <[EMAIL PROTECTED]
> >wrote:
> >
> >> On Thu, Sep 13, 2012 at 7:58 AM, n keywal <[EMAIL PROTECTED]> wrote:
> >> > Hi,
> >> >
> >> > Is there any known counter indication for read short circuit?
> >> > I had a look at JD's presentation from January. It was tested vs.
> HBase
> >> as
> >> > a whole; and still recommended for nearly all cases.
> >>
> >> Pretty much.
> >>
> >> >
> >> > We could make it at least the default when reading HLog files (even
> if it
> >> > seems it's a 'per client" config, there is may be something to look
> at as
> >> > well)?
> >>
> >> You have to add 2 configs in hdfs-site.xml, which we can't do from
> HBase.
> >>
> >> J-D
> >>
>
>
>
> --
> Best regards,
>
> - Andy
>
> Problems worthy of attack prove their worth by hitting back. - Piet
> Hein (via Tom White)
>
-
Re: read short circuitJean-Daniel Cryans 2012-09-13, 18:01
On Thu, Sep 13, 2012 at 10:28 AM, Stack <[EMAIL PROTECTED]> wrote:
> Write a short paragraph and I'll make an HDFS configuration sections
> like this HBase configurations section on manual and stick it in
> there: http://hbase.apache.org/book.html#perf.configurations
Here's a first stab:
Leveraging local data
Since Hadoop 1.0.0 (also 0.22.1, 0.23.1, CDH3u3 and HDP 1.0) via
HDFS-2246[1], it is possible for the DFSClient to take a shortcut and
read directly from disk instead of going through the DataNode when the
data is local. What this means for HBase is that the RegionServers can
read directly off their machine's disks instead of having to open a
socket to talk to the DataNode, the former being generally much
faster[2]. In order to enable it, first hdfs-site.xml needs to be
amended with:
dfs.block.local-path-access.user = the _only_ user that can use the
shortcut. This has to be the user that started HBase.
And in hbase-site.xml:
dfs.client.read.shortcircuit = true
The DataNodes need to be restarted in order to pick up the new
configuration. Be aware that if a process started under another
username than the one configured here also has the shortcircuit
enabled, it will get an Exception regarding an unauthorized access but
the data will still be read.
1. https://issues.apache.org/jira/browse/HDFS-2246
2. http://files.meetup.com/1350427/hug_ebay_jdcryans.pdf
> Write a short paragraph and I'll make an HDFS configuration sections
> like this HBase configurations section on manual and stick it in
> there: http://hbase.apache.org/book.html#perf.configurations
Here's a first stab:
Leveraging local data
Since Hadoop 1.0.0 (also 0.22.1, 0.23.1, CDH3u3 and HDP 1.0) via
HDFS-2246[1], it is possible for the DFSClient to take a shortcut and
read directly from disk instead of going through the DataNode when the
data is local. What this means for HBase is that the RegionServers can
read directly off their machine's disks instead of having to open a
socket to talk to the DataNode, the former being generally much
faster[2]. In order to enable it, first hdfs-site.xml needs to be
amended with:
dfs.block.local-path-access.user = the _only_ user that can use the
shortcut. This has to be the user that started HBase.
And in hbase-site.xml:
dfs.client.read.shortcircuit = true
The DataNodes need to be restarted in order to pick up the new
configuration. Be aware that if a process started under another
username than the one configured here also has the shortcircuit
enabled, it will get an Exception regarding an unauthorized access but
the data will still be read.
1. https://issues.apache.org/jira/browse/HDFS-2246
2. http://files.meetup.com/1350427/hug_ebay_jdcryans.pdf
-
Re: read short circuitAndrew Purtell 2012-09-13, 18:31
That's https://issues.apache.org/jira/browse/HBASE-4867
On Thu, Sep 13, 2012 at 10:57 AM, Jesse Yates <[EMAIL PROTECTED]> wrote:
> There was a jira a while back (can't remember the number) that I think came
> from the Facebook fellas to have multiple hbase-site.xmls, with a naming
> scheme such that you can do layering of configurations.
>
> What ever happened to that?
> -------------------
> Jesse Yates
> @jesse_yates
> jyates.github.com
>
>
> On Thu, Sep 13, 2012 at 10:44 AM, Andrew Purtell <[EMAIL PROTECTED]>wrote:
>
>> Making it the default seems reasonable.
>>
>> For <= 0.94, the security profile has a separate hbase-site.xml file
>> under test resources where there it could be defaulted as off. Reason
>> is the local block sharing requires some shared group membership
>> between HBase and HDFS service accounts that might be objectionable in
>> a secure setting. So we should have test coverage with it off.
>>
>> On Thu, Sep 13, 2012 at 9:42 AM, n keywal <[EMAIL PROTECTED]> wrote:
>> > It's all in the "pretty much" :-)
>> > Should we make it the default (i.e. recommending it in the doc, emitting
>> a
>> > warning if it's not set) for all installation ?
>> > Or would it make sense to separate the store and wal cases, meaning that
>> we
>> > need a change in hdfs to make this possible?
>> >
>> >
>> > On Thu, Sep 13, 2012 at 6:30 PM, Jean-Daniel Cryans <[EMAIL PROTECTED]
>> >wrote:
>> >
>> >> On Thu, Sep 13, 2012 at 7:58 AM, n keywal <[EMAIL PROTECTED]> wrote:
>> >> > Hi,
>> >> >
>> >> > Is there any known counter indication for read short circuit?
>> >> > I had a look at JD's presentation from January. It was tested vs.
>> HBase
>> >> as
>> >> > a whole; and still recommended for nearly all cases.
>> >>
>> >> Pretty much.
>> >>
>> >> >
>> >> > We could make it at least the default when reading HLog files (even
>> if it
>> >> > seems it's a 'per client" config, there is may be something to look
>> at as
>> >> > well)?
>> >>
>> >> You have to add 2 configs in hdfs-site.xml, which we can't do from
>> HBase.
>> >>
>> >> J-D
>> >>
>>
>>
>>
>> --
>> Best regards,
>>
>> - Andy
>>
>> Problems worthy of attack prove their worth by hitting back. - Piet
>> Hein (via Tom White)
>>
--
Best regards,
- Andy
Problems worthy of attack prove their worth by hitting back. - Piet
Hein (via Tom White)
On Thu, Sep 13, 2012 at 10:57 AM, Jesse Yates <[EMAIL PROTECTED]> wrote:
> There was a jira a while back (can't remember the number) that I think came
> from the Facebook fellas to have multiple hbase-site.xmls, with a naming
> scheme such that you can do layering of configurations.
>
> What ever happened to that?
> -------------------
> Jesse Yates
> @jesse_yates
> jyates.github.com
>
>
> On Thu, Sep 13, 2012 at 10:44 AM, Andrew Purtell <[EMAIL PROTECTED]>wrote:
>
>> Making it the default seems reasonable.
>>
>> For <= 0.94, the security profile has a separate hbase-site.xml file
>> under test resources where there it could be defaulted as off. Reason
>> is the local block sharing requires some shared group membership
>> between HBase and HDFS service accounts that might be objectionable in
>> a secure setting. So we should have test coverage with it off.
>>
>> On Thu, Sep 13, 2012 at 9:42 AM, n keywal <[EMAIL PROTECTED]> wrote:
>> > It's all in the "pretty much" :-)
>> > Should we make it the default (i.e. recommending it in the doc, emitting
>> a
>> > warning if it's not set) for all installation ?
>> > Or would it make sense to separate the store and wal cases, meaning that
>> we
>> > need a change in hdfs to make this possible?
>> >
>> >
>> > On Thu, Sep 13, 2012 at 6:30 PM, Jean-Daniel Cryans <[EMAIL PROTECTED]
>> >wrote:
>> >
>> >> On Thu, Sep 13, 2012 at 7:58 AM, n keywal <[EMAIL PROTECTED]> wrote:
>> >> > Hi,
>> >> >
>> >> > Is there any known counter indication for read short circuit?
>> >> > I had a look at JD's presentation from January. It was tested vs.
>> HBase
>> >> as
>> >> > a whole; and still recommended for nearly all cases.
>> >>
>> >> Pretty much.
>> >>
>> >> >
>> >> > We could make it at least the default when reading HLog files (even
>> if it
>> >> > seems it's a 'per client" config, there is may be something to look
>> at as
>> >> > well)?
>> >>
>> >> You have to add 2 configs in hdfs-site.xml, which we can't do from
>> HBase.
>> >>
>> >> J-D
>> >>
>>
>>
>>
>> --
>> Best regards,
>>
>> - Andy
>>
>> Problems worthy of attack prove their worth by hitting back. - Piet
>> Hein (via Tom White)
>>
--
Best regards,
- Andy
Problems worthy of attack prove their worth by hitting back. - Piet
Hein (via Tom White)
-
Re: read short circuitn keywal 2012-09-13, 19:17
If I sum-up :
We make it the default:
1) Documented as such (what JD wrote already + a reference to HDFS-3907)
2) Set in the config we use for the unit tests.
Andrew, you said that we should keep some test with it off ? May be we
could do that in one of the existing build on the trunk, for example
HBase-TRUNK-on-Hadoop-2.0.0? i.e.: We run the tests on trunk with
short-circuit on; we run the test on hadoop-2 with short circuit off.
I propose it this way because may be the tests will be faster with
short-circuit on, and as well if it's the default production config it
should be the one in the "main" test path.
If that's ok, I can do test it on our test suite and do the jira for the
integration.
On Thu, Sep 13, 2012 at 8:31 PM, Andrew Purtell <[EMAIL PROTECTED]> wrote:
> That's https://issues.apache.org/jira/browse/HBASE-4867
>
> On Thu, Sep 13, 2012 at 10:57 AM, Jesse Yates <[EMAIL PROTECTED]>
> wrote:
> > There was a jira a while back (can't remember the number) that I think
> came
> > from the Facebook fellas to have multiple hbase-site.xmls, with a naming
> > scheme such that you can do layering of configurations.
> >
> > What ever happened to that?
> > -------------------
> > Jesse Yates
> > @jesse_yates
> > jyates.github.com
> >
> >
> > On Thu, Sep 13, 2012 at 10:44 AM, Andrew Purtell <[EMAIL PROTECTED]
> >wrote:
> >
> >> Making it the default seems reasonable.
> >>
> >> For <= 0.94, the security profile has a separate hbase-site.xml file
> >> under test resources where there it could be defaulted as off. Reason
> >> is the local block sharing requires some shared group membership
> >> between HBase and HDFS service accounts that might be objectionable in
> >> a secure setting. So we should have test coverage with it off.
> >>
> >> On Thu, Sep 13, 2012 at 9:42 AM, n keywal <[EMAIL PROTECTED]> wrote:
> >> > It's all in the "pretty much" :-)
> >> > Should we make it the default (i.e. recommending it in the doc,
> emitting
> >> a
> >> > warning if it's not set) for all installation ?
> >> > Or would it make sense to separate the store and wal cases, meaning
> that
> >> we
> >> > need a change in hdfs to make this possible?
> >> >
> >> >
> >> > On Thu, Sep 13, 2012 at 6:30 PM, Jean-Daniel Cryans <
> [EMAIL PROTECTED]
> >> >wrote:
> >> >
> >> >> On Thu, Sep 13, 2012 at 7:58 AM, n keywal <[EMAIL PROTECTED]> wrote:
> >> >> > Hi,
> >> >> >
> >> >> > Is there any known counter indication for read short circuit?
> >> >> > I had a look at JD's presentation from January. It was tested vs.
> >> HBase
> >> >> as
> >> >> > a whole; and still recommended for nearly all cases.
> >> >>
> >> >> Pretty much.
> >> >>
> >> >> >
> >> >> > We could make it at least the default when reading HLog files (even
> >> if it
> >> >> > seems it's a 'per client" config, there is may be something to look
> >> at as
> >> >> > well)?
> >> >>
> >> >> You have to add 2 configs in hdfs-site.xml, which we can't do from
> >> HBase.
> >> >>
> >> >> J-D
> >> >>
> >>
> >>
> >>
> >> --
> >> Best regards,
> >>
> >> - Andy
> >>
> >> Problems worthy of attack prove their worth by hitting back. - Piet
> >> Hein (via Tom White)
> >>
>
>
>
> --
> Best regards,
>
> - Andy
>
> Problems worthy of attack prove their worth by hitting back. - Piet
> Hein (via Tom White)
>
We make it the default:
1) Documented as such (what JD wrote already + a reference to HDFS-3907)
2) Set in the config we use for the unit tests.
Andrew, you said that we should keep some test with it off ? May be we
could do that in one of the existing build on the trunk, for example
HBase-TRUNK-on-Hadoop-2.0.0? i.e.: We run the tests on trunk with
short-circuit on; we run the test on hadoop-2 with short circuit off.
I propose it this way because may be the tests will be faster with
short-circuit on, and as well if it's the default production config it
should be the one in the "main" test path.
If that's ok, I can do test it on our test suite and do the jira for the
integration.
On Thu, Sep 13, 2012 at 8:31 PM, Andrew Purtell <[EMAIL PROTECTED]> wrote:
> That's https://issues.apache.org/jira/browse/HBASE-4867
>
> On Thu, Sep 13, 2012 at 10:57 AM, Jesse Yates <[EMAIL PROTECTED]>
> wrote:
> > There was a jira a while back (can't remember the number) that I think
> came
> > from the Facebook fellas to have multiple hbase-site.xmls, with a naming
> > scheme such that you can do layering of configurations.
> >
> > What ever happened to that?
> > -------------------
> > Jesse Yates
> > @jesse_yates
> > jyates.github.com
> >
> >
> > On Thu, Sep 13, 2012 at 10:44 AM, Andrew Purtell <[EMAIL PROTECTED]
> >wrote:
> >
> >> Making it the default seems reasonable.
> >>
> >> For <= 0.94, the security profile has a separate hbase-site.xml file
> >> under test resources where there it could be defaulted as off. Reason
> >> is the local block sharing requires some shared group membership
> >> between HBase and HDFS service accounts that might be objectionable in
> >> a secure setting. So we should have test coverage with it off.
> >>
> >> On Thu, Sep 13, 2012 at 9:42 AM, n keywal <[EMAIL PROTECTED]> wrote:
> >> > It's all in the "pretty much" :-)
> >> > Should we make it the default (i.e. recommending it in the doc,
> emitting
> >> a
> >> > warning if it's not set) for all installation ?
> >> > Or would it make sense to separate the store and wal cases, meaning
> that
> >> we
> >> > need a change in hdfs to make this possible?
> >> >
> >> >
> >> > On Thu, Sep 13, 2012 at 6:30 PM, Jean-Daniel Cryans <
> [EMAIL PROTECTED]
> >> >wrote:
> >> >
> >> >> On Thu, Sep 13, 2012 at 7:58 AM, n keywal <[EMAIL PROTECTED]> wrote:
> >> >> > Hi,
> >> >> >
> >> >> > Is there any known counter indication for read short circuit?
> >> >> > I had a look at JD's presentation from January. It was tested vs.
> >> HBase
> >> >> as
> >> >> > a whole; and still recommended for nearly all cases.
> >> >>
> >> >> Pretty much.
> >> >>
> >> >> >
> >> >> > We could make it at least the default when reading HLog files (even
> >> if it
> >> >> > seems it's a 'per client" config, there is may be something to look
> >> at as
> >> >> > well)?
> >> >>
> >> >> You have to add 2 configs in hdfs-site.xml, which we can't do from
> >> HBase.
> >> >>
> >> >> J-D
> >> >>
> >>
> >>
> >>
> >> --
> >> Best regards,
> >>
> >> - Andy
> >>
> >> Problems worthy of attack prove their worth by hitting back. - Piet
> >> Hein (via Tom White)
> >>
>
>
>
> --
> Best regards,
>
> - Andy
>
> Problems worthy of attack prove their worth by hitting back. - Piet
> Hein (via Tom White)
>
-
Re: read short circuitAndrew Purtell 2012-09-13, 20:20
Hey N,
It makes sense to have a 0.94 build with -Psecurity enabled and local
shortcut disabled. What do you think? For trunk builds your points sound
good, and I have no opinion.
On Thursday, September 13, 2012, n keywal wrote:
> If I sum-up :
>
> We make it the default:
> 1) Documented as such (what JD wrote already + a reference to HDFS-3907)
> 2) Set in the config we use for the unit tests.
>
> Andrew, you said that we should keep some test with it off ? May be we
> could do that in one of the existing build on the trunk, for example
> HBase-TRUNK-on-Hadoop-2.0.0? i.e.: We run the tests on trunk with
> short-circuit on; we run the test on hadoop-2 with short circuit off.
>
> I propose it this way because may be the tests will be faster with
> short-circuit on, and as well if it's the default production config it
> should be the one in the "main" test path.
>
> If that's ok, I can do test it on our test suite and do the jira for the
> integration.
>
>
>
>
> On Thu, Sep 13, 2012 at 8:31 PM, Andrew Purtell <[EMAIL PROTECTED]<javascript:;>>
> wrote:
>
> > That's https://issues.apache.org/jira/browse/HBASE-4867
> >
> > On Thu, Sep 13, 2012 at 10:57 AM, Jesse Yates <[EMAIL PROTECTED]<javascript:;>
> >
> > wrote:
> > > There was a jira a while back (can't remember the number) that I think
> > came
> > > from the Facebook fellas to have multiple hbase-site.xmls, with a
> naming
> > > scheme such that you can do layering of configurations.
> > >
> > > What ever happened to that?
> > > -------------------
> > > Jesse Yates
> > > @jesse_yates
> > > jyates.github.com
> > >
> > >
> > > On Thu, Sep 13, 2012 at 10:44 AM, Andrew Purtell <[EMAIL PROTECTED]<javascript:;>
> > >wrote:
> > >
> > >> Making it the default seems reasonable.
> > >>
> > >> For <= 0.94, the security profile has a separate hbase-site.xml file
> > >> under test resources where there it could be defaulted as off. Reason
> > >> is the local block sharing requires some shared group membership
> > >> between HBase and HDFS service accounts that might be objectionable in
> > >> a secure setting. So we should have test coverage with it off.
> > >>
> > >> On Thu, Sep 13, 2012 at 9:42 AM, n keywal <[EMAIL PROTECTED]<javascript:;>>
> wrote:
> > >> > It's all in the "pretty much" :-)
> > >> > Should we make it the default (i.e. recommending it in the doc,
> > emitting
> > >> a
> > >> > warning if it's not set) for all installation ?
> > >> > Or would it make sense to separate the store and wal cases, meaning
> > that
> > >> we
> > >> > need a change in hdfs to make this possible?
> > >> >
> > >> >
> > >> > On Thu, Sep 13, 2012 at 6:30 PM, Jean-Daniel Cryans <
> > [EMAIL PROTECTED] <javascript:;>
> > >> >wrote:
> > >> >
> > >> >> On Thu, Sep 13, 2012 at 7:58 AM, n keywal <[EMAIL PROTECTED]<javascript:;>>
> wrote:
> > >> >> > Hi,
> > >> >> >
> > >> >> > Is there any known counter indication for read short circuit?
> > >> >> > I had a look at JD's presentation from January. It was tested vs.
> > >> HBase
> > >> >> as
> > >> >> > a whole; and still recommended for nearly all cases.
> > >> >>
> > >> >> Pretty much.
> > >> >>
> > >> >> >
> > >> >> > We could make it at least the default when reading HLog files
> (even
> > >> if it
> > >> >> > seems it's a 'per client" config, there is may be something to
> look
> > >> at as
> > >> >> > well)?
> > >> >>
> > >> >> You have to add 2 configs in hdfs-site.xml, which we can't do from
> > >> HBase.
> > >> >>
> > >> >> J-D
> > >> >>
> > >>
> > >>
> > >>
> > >> --
> > >> Best regards,
> > >>
> > >> - Andy
> > >>
> > >> Problems worthy of attack prove their worth by hitting back. - Piet
> > >> Hein (via Tom White)
> > >>
> >
> >
> >
> > --
> > Best regards,
> >
> > - Andy
> >
> > Problems worthy of attack prove their worth by hitting back. - Piet
> > Hein (via Tom White)
> >
>
--
Best regards,
- Andy
Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)
It makes sense to have a 0.94 build with -Psecurity enabled and local
shortcut disabled. What do you think? For trunk builds your points sound
good, and I have no opinion.
On Thursday, September 13, 2012, n keywal wrote:
> If I sum-up :
>
> We make it the default:
> 1) Documented as such (what JD wrote already + a reference to HDFS-3907)
> 2) Set in the config we use for the unit tests.
>
> Andrew, you said that we should keep some test with it off ? May be we
> could do that in one of the existing build on the trunk, for example
> HBase-TRUNK-on-Hadoop-2.0.0? i.e.: We run the tests on trunk with
> short-circuit on; we run the test on hadoop-2 with short circuit off.
>
> I propose it this way because may be the tests will be faster with
> short-circuit on, and as well if it's the default production config it
> should be the one in the "main" test path.
>
> If that's ok, I can do test it on our test suite and do the jira for the
> integration.
>
>
>
>
> On Thu, Sep 13, 2012 at 8:31 PM, Andrew Purtell <[EMAIL PROTECTED]<javascript:;>>
> wrote:
>
> > That's https://issues.apache.org/jira/browse/HBASE-4867
> >
> > On Thu, Sep 13, 2012 at 10:57 AM, Jesse Yates <[EMAIL PROTECTED]<javascript:;>
> >
> > wrote:
> > > There was a jira a while back (can't remember the number) that I think
> > came
> > > from the Facebook fellas to have multiple hbase-site.xmls, with a
> naming
> > > scheme such that you can do layering of configurations.
> > >
> > > What ever happened to that?
> > > -------------------
> > > Jesse Yates
> > > @jesse_yates
> > > jyates.github.com
> > >
> > >
> > > On Thu, Sep 13, 2012 at 10:44 AM, Andrew Purtell <[EMAIL PROTECTED]<javascript:;>
> > >wrote:
> > >
> > >> Making it the default seems reasonable.
> > >>
> > >> For <= 0.94, the security profile has a separate hbase-site.xml file
> > >> under test resources where there it could be defaulted as off. Reason
> > >> is the local block sharing requires some shared group membership
> > >> between HBase and HDFS service accounts that might be objectionable in
> > >> a secure setting. So we should have test coverage with it off.
> > >>
> > >> On Thu, Sep 13, 2012 at 9:42 AM, n keywal <[EMAIL PROTECTED]<javascript:;>>
> wrote:
> > >> > It's all in the "pretty much" :-)
> > >> > Should we make it the default (i.e. recommending it in the doc,
> > emitting
> > >> a
> > >> > warning if it's not set) for all installation ?
> > >> > Or would it make sense to separate the store and wal cases, meaning
> > that
> > >> we
> > >> > need a change in hdfs to make this possible?
> > >> >
> > >> >
> > >> > On Thu, Sep 13, 2012 at 6:30 PM, Jean-Daniel Cryans <
> > [EMAIL PROTECTED] <javascript:;>
> > >> >wrote:
> > >> >
> > >> >> On Thu, Sep 13, 2012 at 7:58 AM, n keywal <[EMAIL PROTECTED]<javascript:;>>
> wrote:
> > >> >> > Hi,
> > >> >> >
> > >> >> > Is there any known counter indication for read short circuit?
> > >> >> > I had a look at JD's presentation from January. It was tested vs.
> > >> HBase
> > >> >> as
> > >> >> > a whole; and still recommended for nearly all cases.
> > >> >>
> > >> >> Pretty much.
> > >> >>
> > >> >> >
> > >> >> > We could make it at least the default when reading HLog files
> (even
> > >> if it
> > >> >> > seems it's a 'per client" config, there is may be something to
> look
> > >> at as
> > >> >> > well)?
> > >> >>
> > >> >> You have to add 2 configs in hdfs-site.xml, which we can't do from
> > >> HBase.
> > >> >>
> > >> >> J-D
> > >> >>
> > >>
> > >>
> > >>
> > >> --
> > >> Best regards,
> > >>
> > >> - Andy
> > >>
> > >> Problems worthy of attack prove their worth by hitting back. - Piet
> > >> Hein (via Tom White)
> > >>
> >
> >
> >
> > --
> > Best regards,
> >
> > - Andy
> >
> > Problems worthy of attack prove their worth by hitting back. - Piet
> > Hein (via Tom White)
> >
>
--
Best regards,
- Andy
Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)
-
Re: read short circuitn keywal 2012-09-13, 20:34
We've got two security builds on 0.94: HBase-0.94-security and
HBase-0.94-security-on-Hadoop-23
We can change the setting on one of them?
On Thu, Sep 13, 2012 at 10:20 PM, Andrew Purtell <[EMAIL PROTECTED]>wrote:
> Hey N,
>
> It makes sense to have a 0.94 build with -Psecurity enabled and local
> shortcut disabled. What do you think? For trunk builds your points sound
> good, and I have no opinion.
>
> On Thursday, September 13, 2012, n keywal wrote:
>
> > If I sum-up :
> >
> > We make it the default:
> > 1) Documented as such (what JD wrote already + a reference to HDFS-3907)
> > 2) Set in the config we use for the unit tests.
> >
> > Andrew, you said that we should keep some test with it off ? May be we
> > could do that in one of the existing build on the trunk, for example
> > HBase-TRUNK-on-Hadoop-2.0.0? i.e.: We run the tests on trunk with
> > short-circuit on; we run the test on hadoop-2 with short circuit off.
> >
> > I propose it this way because may be the tests will be faster with
> > short-circuit on, and as well if it's the default production config it
> > should be the one in the "main" test path.
> >
> > If that's ok, I can do test it on our test suite and do the jira for the
> > integration.
> >
> >
> >
> >
> > On Thu, Sep 13, 2012 at 8:31 PM, Andrew Purtell <[EMAIL PROTECTED]
> <javascript:;>>
> > wrote:
> >
> > > That's https://issues.apache.org/jira/browse/HBASE-4867
> > >
> > > On Thu, Sep 13, 2012 at 10:57 AM, Jesse Yates <[EMAIL PROTECTED]
> <javascript:;>
> > >
> > > wrote:
> > > > There was a jira a while back (can't remember the number) that I
> think
> > > came
> > > > from the Facebook fellas to have multiple hbase-site.xmls, with a
> > naming
> > > > scheme such that you can do layering of configurations.
> > > >
> > > > What ever happened to that?
> > > > -------------------
> > > > Jesse Yates
> > > > @jesse_yates
> > > > jyates.github.com
> > > >
> > > >
> > > > On Thu, Sep 13, 2012 at 10:44 AM, Andrew Purtell <
> [EMAIL PROTECTED]<javascript:;>
> > > >wrote:
> > > >
> > > >> Making it the default seems reasonable.
> > > >>
> > > >> For <= 0.94, the security profile has a separate hbase-site.xml file
> > > >> under test resources where there it could be defaulted as off.
> Reason
> > > >> is the local block sharing requires some shared group membership
> > > >> between HBase and HDFS service accounts that might be objectionable
> in
> > > >> a secure setting. So we should have test coverage with it off.
> > > >>
> > > >> On Thu, Sep 13, 2012 at 9:42 AM, n keywal <[EMAIL PROTECTED]
> <javascript:;>>
> > wrote:
> > > >> > It's all in the "pretty much" :-)
> > > >> > Should we make it the default (i.e. recommending it in the doc,
> > > emitting
> > > >> a
> > > >> > warning if it's not set) for all installation ?
> > > >> > Or would it make sense to separate the store and wal cases,
> meaning
> > > that
> > > >> we
> > > >> > need a change in hdfs to make this possible?
> > > >> >
> > > >> >
> > > >> > On Thu, Sep 13, 2012 at 6:30 PM, Jean-Daniel Cryans <
> > > [EMAIL PROTECTED] <javascript:;>
> > > >> >wrote:
> > > >> >
> > > >> >> On Thu, Sep 13, 2012 at 7:58 AM, n keywal <[EMAIL PROTECTED]
> <javascript:;>>
> > wrote:
> > > >> >> > Hi,
> > > >> >> >
> > > >> >> > Is there any known counter indication for read short circuit?
> > > >> >> > I had a look at JD's presentation from January. It was tested
> vs.
> > > >> HBase
> > > >> >> as
> > > >> >> > a whole; and still recommended for nearly all cases.
> > > >> >>
> > > >> >> Pretty much.
> > > >> >>
> > > >> >> >
> > > >> >> > We could make it at least the default when reading HLog files
> > (even
> > > >> if it
> > > >> >> > seems it's a 'per client" config, there is may be something to
> > look
> > > >> at as
> > > >> >> > well)?
> > > >> >>
> > > >> >> You have to add 2 configs in hdfs-site.xml, which we can't do
> from
> > > >> HBase.
> > > >> >>
> > > >> >> J-D
> > > >> >>
> > > >>
> > > >>
> > > >>
> > > >> --
> > > >> Best regards,
HBase-0.94-security-on-Hadoop-23
We can change the setting on one of them?
On Thu, Sep 13, 2012 at 10:20 PM, Andrew Purtell <[EMAIL PROTECTED]>wrote:
> Hey N,
>
> It makes sense to have a 0.94 build with -Psecurity enabled and local
> shortcut disabled. What do you think? For trunk builds your points sound
> good, and I have no opinion.
>
> On Thursday, September 13, 2012, n keywal wrote:
>
> > If I sum-up :
> >
> > We make it the default:
> > 1) Documented as such (what JD wrote already + a reference to HDFS-3907)
> > 2) Set in the config we use for the unit tests.
> >
> > Andrew, you said that we should keep some test with it off ? May be we
> > could do that in one of the existing build on the trunk, for example
> > HBase-TRUNK-on-Hadoop-2.0.0? i.e.: We run the tests on trunk with
> > short-circuit on; we run the test on hadoop-2 with short circuit off.
> >
> > I propose it this way because may be the tests will be faster with
> > short-circuit on, and as well if it's the default production config it
> > should be the one in the "main" test path.
> >
> > If that's ok, I can do test it on our test suite and do the jira for the
> > integration.
> >
> >
> >
> >
> > On Thu, Sep 13, 2012 at 8:31 PM, Andrew Purtell <[EMAIL PROTECTED]
> <javascript:;>>
> > wrote:
> >
> > > That's https://issues.apache.org/jira/browse/HBASE-4867
> > >
> > > On Thu, Sep 13, 2012 at 10:57 AM, Jesse Yates <[EMAIL PROTECTED]
> <javascript:;>
> > >
> > > wrote:
> > > > There was a jira a while back (can't remember the number) that I
> think
> > > came
> > > > from the Facebook fellas to have multiple hbase-site.xmls, with a
> > naming
> > > > scheme such that you can do layering of configurations.
> > > >
> > > > What ever happened to that?
> > > > -------------------
> > > > Jesse Yates
> > > > @jesse_yates
> > > > jyates.github.com
> > > >
> > > >
> > > > On Thu, Sep 13, 2012 at 10:44 AM, Andrew Purtell <
> [EMAIL PROTECTED]<javascript:;>
> > > >wrote:
> > > >
> > > >> Making it the default seems reasonable.
> > > >>
> > > >> For <= 0.94, the security profile has a separate hbase-site.xml file
> > > >> under test resources where there it could be defaulted as off.
> Reason
> > > >> is the local block sharing requires some shared group membership
> > > >> between HBase and HDFS service accounts that might be objectionable
> in
> > > >> a secure setting. So we should have test coverage with it off.
> > > >>
> > > >> On Thu, Sep 13, 2012 at 9:42 AM, n keywal <[EMAIL PROTECTED]
> <javascript:;>>
> > wrote:
> > > >> > It's all in the "pretty much" :-)
> > > >> > Should we make it the default (i.e. recommending it in the doc,
> > > emitting
> > > >> a
> > > >> > warning if it's not set) for all installation ?
> > > >> > Or would it make sense to separate the store and wal cases,
> meaning
> > > that
> > > >> we
> > > >> > need a change in hdfs to make this possible?
> > > >> >
> > > >> >
> > > >> > On Thu, Sep 13, 2012 at 6:30 PM, Jean-Daniel Cryans <
> > > [EMAIL PROTECTED] <javascript:;>
> > > >> >wrote:
> > > >> >
> > > >> >> On Thu, Sep 13, 2012 at 7:58 AM, n keywal <[EMAIL PROTECTED]
> <javascript:;>>
> > wrote:
> > > >> >> > Hi,
> > > >> >> >
> > > >> >> > Is there any known counter indication for read short circuit?
> > > >> >> > I had a look at JD's presentation from January. It was tested
> vs.
> > > >> HBase
> > > >> >> as
> > > >> >> > a whole; and still recommended for nearly all cases.
> > > >> >>
> > > >> >> Pretty much.
> > > >> >>
> > > >> >> >
> > > >> >> > We could make it at least the default when reading HLog files
> > (even
> > > >> if it
> > > >> >> > seems it's a 'per client" config, there is may be something to
> > look
> > > >> at as
> > > >> >> > well)?
> > > >> >>
> > > >> >> You have to add 2 configs in hdfs-site.xml, which we can't do
> from
> > > >> HBase.
> > > >> >>
> > > >> >> J-D
> > > >> >>
> > > >>
> > > >>
> > > >>
> > > >> --
> > > >> Best regards,
-
Re: read short circuitAndrew Purtell 2012-09-13, 21:07
Yeah, on HBase-0.94-security
On Thu, Sep 13, 2012 at 1:34 PM, n keywal <[EMAIL PROTECTED]> wrote:
> We've got two security builds on 0.94: HBase-0.94-security and
> HBase-0.94-security-on-Hadoop-23
> We can change the setting on one of them?
>
> On Thu, Sep 13, 2012 at 10:20 PM, Andrew Purtell <[EMAIL PROTECTED]>wrote:
>
>> Hey N,
>>
>> It makes sense to have a 0.94 build with -Psecurity enabled and local
>> shortcut disabled. What do you think? For trunk builds your points sound
>> good, and I have no opinion.
>>
>> On Thursday, September 13, 2012, n keywal wrote:
>>
>> > If I sum-up :
>> >
>> > We make it the default:
>> > 1) Documented as such (what JD wrote already + a reference to HDFS-3907)
>> > 2) Set in the config we use for the unit tests.
>> >
>> > Andrew, you said that we should keep some test with it off ? May be we
>> > could do that in one of the existing build on the trunk, for example
>> > HBase-TRUNK-on-Hadoop-2.0.0? i.e.: We run the tests on trunk with
>> > short-circuit on; we run the test on hadoop-2 with short circuit off.
>> >
>> > I propose it this way because may be the tests will be faster with
>> > short-circuit on, and as well if it's the default production config it
>> > should be the one in the "main" test path.
>> >
>> > If that's ok, I can do test it on our test suite and do the jira for the
>> > integration.
>> >
>> >
>> >
>> >
>> > On Thu, Sep 13, 2012 at 8:31 PM, Andrew Purtell <[EMAIL PROTECTED]
>> <javascript:;>>
>> > wrote:
>> >
>> > > That's https://issues.apache.org/jira/browse/HBASE-4867
>> > >
>> > > On Thu, Sep 13, 2012 at 10:57 AM, Jesse Yates <[EMAIL PROTECTED]
>> <javascript:;>
>> > >
>> > > wrote:
>> > > > There was a jira a while back (can't remember the number) that I
>> think
>> > > came
>> > > > from the Facebook fellas to have multiple hbase-site.xmls, with a
>> > naming
>> > > > scheme such that you can do layering of configurations.
>> > > >
>> > > > What ever happened to that?
>> > > > -------------------
>> > > > Jesse Yates
>> > > > @jesse_yates
>> > > > jyates.github.com
>> > > >
>> > > >
>> > > > On Thu, Sep 13, 2012 at 10:44 AM, Andrew Purtell <
>> [EMAIL PROTECTED]<javascript:;>
>> > > >wrote:
>> > > >
>> > > >> Making it the default seems reasonable.
>> > > >>
>> > > >> For <= 0.94, the security profile has a separate hbase-site.xml file
>> > > >> under test resources where there it could be defaulted as off.
>> Reason
>> > > >> is the local block sharing requires some shared group membership
>> > > >> between HBase and HDFS service accounts that might be objectionable
>> in
>> > > >> a secure setting. So we should have test coverage with it off.
>> > > >>
>> > > >> On Thu, Sep 13, 2012 at 9:42 AM, n keywal <[EMAIL PROTECTED]
>> <javascript:;>>
>> > wrote:
>> > > >> > It's all in the "pretty much" :-)
>> > > >> > Should we make it the default (i.e. recommending it in the doc,
>> > > emitting
>> > > >> a
>> > > >> > warning if it's not set) for all installation ?
>> > > >> > Or would it make sense to separate the store and wal cases,
>> meaning
>> > > that
>> > > >> we
>> > > >> > need a change in hdfs to make this possible?
>> > > >> >
>> > > >> >
>> > > >> > On Thu, Sep 13, 2012 at 6:30 PM, Jean-Daniel Cryans <
>> > > [EMAIL PROTECTED] <javascript:;>
>> > > >> >wrote:
>> > > >> >
>> > > >> >> On Thu, Sep 13, 2012 at 7:58 AM, n keywal <[EMAIL PROTECTED]
>> <javascript:;>>
>> > wrote:
>> > > >> >> > Hi,
>> > > >> >> >
>> > > >> >> > Is there any known counter indication for read short circuit?
>> > > >> >> > I had a look at JD's presentation from January. It was tested
>> vs.
>> > > >> HBase
>> > > >> >> as
>> > > >> >> > a whole; and still recommended for nearly all cases.
>> > > >> >>
>> > > >> >> Pretty much.
>> > > >> >>
>> > > >> >> >
>> > > >> >> > We could make it at least the default when reading HLog files
>> > (even
>> > > >> if it
>> > > >> >> > seems it's a 'per client" config, there is may be something to
>> > look
>> > > >> at as
>> > > >
Best regards,
- Andy
Problems worthy of attack prove their worth by hitting back. - Piet
Hein (via Tom White)
On Thu, Sep 13, 2012 at 1:34 PM, n keywal <[EMAIL PROTECTED]> wrote:
> We've got two security builds on 0.94: HBase-0.94-security and
> HBase-0.94-security-on-Hadoop-23
> We can change the setting on one of them?
>
> On Thu, Sep 13, 2012 at 10:20 PM, Andrew Purtell <[EMAIL PROTECTED]>wrote:
>
>> Hey N,
>>
>> It makes sense to have a 0.94 build with -Psecurity enabled and local
>> shortcut disabled. What do you think? For trunk builds your points sound
>> good, and I have no opinion.
>>
>> On Thursday, September 13, 2012, n keywal wrote:
>>
>> > If I sum-up :
>> >
>> > We make it the default:
>> > 1) Documented as such (what JD wrote already + a reference to HDFS-3907)
>> > 2) Set in the config we use for the unit tests.
>> >
>> > Andrew, you said that we should keep some test with it off ? May be we
>> > could do that in one of the existing build on the trunk, for example
>> > HBase-TRUNK-on-Hadoop-2.0.0? i.e.: We run the tests on trunk with
>> > short-circuit on; we run the test on hadoop-2 with short circuit off.
>> >
>> > I propose it this way because may be the tests will be faster with
>> > short-circuit on, and as well if it's the default production config it
>> > should be the one in the "main" test path.
>> >
>> > If that's ok, I can do test it on our test suite and do the jira for the
>> > integration.
>> >
>> >
>> >
>> >
>> > On Thu, Sep 13, 2012 at 8:31 PM, Andrew Purtell <[EMAIL PROTECTED]
>> <javascript:;>>
>> > wrote:
>> >
>> > > That's https://issues.apache.org/jira/browse/HBASE-4867
>> > >
>> > > On Thu, Sep 13, 2012 at 10:57 AM, Jesse Yates <[EMAIL PROTECTED]
>> <javascript:;>
>> > >
>> > > wrote:
>> > > > There was a jira a while back (can't remember the number) that I
>> think
>> > > came
>> > > > from the Facebook fellas to have multiple hbase-site.xmls, with a
>> > naming
>> > > > scheme such that you can do layering of configurations.
>> > > >
>> > > > What ever happened to that?
>> > > > -------------------
>> > > > Jesse Yates
>> > > > @jesse_yates
>> > > > jyates.github.com
>> > > >
>> > > >
>> > > > On Thu, Sep 13, 2012 at 10:44 AM, Andrew Purtell <
>> [EMAIL PROTECTED]<javascript:;>
>> > > >wrote:
>> > > >
>> > > >> Making it the default seems reasonable.
>> > > >>
>> > > >> For <= 0.94, the security profile has a separate hbase-site.xml file
>> > > >> under test resources where there it could be defaulted as off.
>> Reason
>> > > >> is the local block sharing requires some shared group membership
>> > > >> between HBase and HDFS service accounts that might be objectionable
>> in
>> > > >> a secure setting. So we should have test coverage with it off.
>> > > >>
>> > > >> On Thu, Sep 13, 2012 at 9:42 AM, n keywal <[EMAIL PROTECTED]
>> <javascript:;>>
>> > wrote:
>> > > >> > It's all in the "pretty much" :-)
>> > > >> > Should we make it the default (i.e. recommending it in the doc,
>> > > emitting
>> > > >> a
>> > > >> > warning if it's not set) for all installation ?
>> > > >> > Or would it make sense to separate the store and wal cases,
>> meaning
>> > > that
>> > > >> we
>> > > >> > need a change in hdfs to make this possible?
>> > > >> >
>> > > >> >
>> > > >> > On Thu, Sep 13, 2012 at 6:30 PM, Jean-Daniel Cryans <
>> > > [EMAIL PROTECTED] <javascript:;>
>> > > >> >wrote:
>> > > >> >
>> > > >> >> On Thu, Sep 13, 2012 at 7:58 AM, n keywal <[EMAIL PROTECTED]
>> <javascript:;>>
>> > wrote:
>> > > >> >> > Hi,
>> > > >> >> >
>> > > >> >> > Is there any known counter indication for read short circuit?
>> > > >> >> > I had a look at JD's presentation from January. It was tested
>> vs.
>> > > >> HBase
>> > > >> >> as
>> > > >> >> > a whole; and still recommended for nearly all cases.
>> > > >> >>
>> > > >> >> Pretty much.
>> > > >> >>
>> > > >> >> >
>> > > >> >> > We could make it at least the default when reading HLog files
>> > (even
>> > > >> if it
>> > > >> >> > seems it's a 'per client" config, there is may be something to
>> > look
>> > > >> at as
>> > > >
Best regards,
- Andy
Problems worthy of attack prove their worth by hitting back. - Piet
Hein (via Tom White)
-
Re: read short circuitn keywal 2012-09-14, 12:48
Done for trunk on HBASE-6783. I will need to modify the tests script to
unactivate it on HBase-0.94-security and HBase-TRUNK-on-Hadoop-2.0.0.
As a side note, there is a difference in behavior between the two modes: we
don't see anymore a datanode error. I've seen it mentioned by JD somewhere,
and I remember that the conclusion was "it's ok". Can't find it again,
though.
Conclusion; the TestFSErrorsExposed#testFullSystemBubblesFSErrors is not
executed if short circuit is on.
On Thu, Sep 13, 2012 at 11:07 PM, Andrew Purtell <[EMAIL PROTECTED]>wrote:
> Yeah, on HBase-0.94-security
>
> On Thu, Sep 13, 2012 at 1:34 PM, n keywal <[EMAIL PROTECTED]> wrote:
> > We've got two security builds on 0.94: HBase-0.94-security and
> > HBase-0.94-security-on-Hadoop-23
> > We can change the setting on one of them?
> >
> > On Thu, Sep 13, 2012 at 10:20 PM, Andrew Purtell <[EMAIL PROTECTED]
> >wrote:
> >
> >> Hey N,
> >>
> >> It makes sense to have a 0.94 build with -Psecurity enabled and local
> >> shortcut disabled. What do you think? For trunk builds your points sound
> >> good, and I have no opinion.
> >>
> >> On Thursday, September 13, 2012, n keywal wrote:
> >>
> >> > If I sum-up :
> >> >
> >> > We make it the default:
> >> > 1) Documented as such (what JD wrote already + a reference to
> HDFS-3907)
> >> > 2) Set in the config we use for the unit tests.
> >> >
> >> > Andrew, you said that we should keep some test with it off ? May be we
> >> > could do that in one of the existing build on the trunk, for example
> >> > HBase-TRUNK-on-Hadoop-2.0.0? i.e.: We run the tests on trunk with
> >> > short-circuit on; we run the test on hadoop-2 with short circuit off.
> >> >
> >> > I propose it this way because may be the tests will be faster with
> >> > short-circuit on, and as well if it's the default production config it
> >> > should be the one in the "main" test path.
> >> >
> >> > If that's ok, I can do test it on our test suite and do the jira for
> the
> >> > integration.
> >> >
> >> >
> >> >
> >> >
> >> > On Thu, Sep 13, 2012 at 8:31 PM, Andrew Purtell <[EMAIL PROTECTED]
> >> <javascript:;>>
> >> > wrote:
> >> >
> >> > > That's https://issues.apache.org/jira/browse/HBASE-4867
> >> > >
> >> > > On Thu, Sep 13, 2012 at 10:57 AM, Jesse Yates <
> [EMAIL PROTECTED]
> >> <javascript:;>
> >> > >
> >> > > wrote:
> >> > > > There was a jira a while back (can't remember the number) that I
> >> think
> >> > > came
> >> > > > from the Facebook fellas to have multiple hbase-site.xmls, with a
> >> > naming
> >> > > > scheme such that you can do layering of configurations.
> >> > > >
> >> > > > What ever happened to that?
> >> > > > -------------------
> >> > > > Jesse Yates
> >> > > > @jesse_yates
> >> > > > jyates.github.com
> >> > > >
> >> > > >
> >> > > > On Thu, Sep 13, 2012 at 10:44 AM, Andrew Purtell <
> >> [EMAIL PROTECTED]<javascript:;>
> >> > > >wrote:
> >> > > >
> >> > > >> Making it the default seems reasonable.
> >> > > >>
> >> > > >> For <= 0.94, the security profile has a separate hbase-site.xml
> file
> >> > > >> under test resources where there it could be defaulted as off.
> >> Reason
> >> > > >> is the local block sharing requires some shared group membership
> >> > > >> between HBase and HDFS service accounts that might be
> objectionable
> >> in
> >> > > >> a secure setting. So we should have test coverage with it off.
> >> > > >>
> >> > > >> On Thu, Sep 13, 2012 at 9:42 AM, n keywal <[EMAIL PROTECTED]
> >> <javascript:;>>
> >> > wrote:
> >> > > >> > It's all in the "pretty much" :-)
> >> > > >> > Should we make it the default (i.e. recommending it in the doc,
> >> > > emitting
> >> > > >> a
> >> > > >> > warning if it's not set) for all installation ?
> >> > > >> > Or would it make sense to separate the store and wal cases,
> >> meaning
> >> > > that
> >> > > >> we
> >> > > >> > need a change in hdfs to make this possible?
> >> > > >> >
> >> > > >> >
> >> > > >> > On Thu, Sep 13, 2012 at 6:30 PM, Jean-Daniel Cryans <
unactivate it on HBase-0.94-security and HBase-TRUNK-on-Hadoop-2.0.0.
As a side note, there is a difference in behavior between the two modes: we
don't see anymore a datanode error. I've seen it mentioned by JD somewhere,
and I remember that the conclusion was "it's ok". Can't find it again,
though.
Conclusion; the TestFSErrorsExposed#testFullSystemBubblesFSErrors is not
executed if short circuit is on.
On Thu, Sep 13, 2012 at 11:07 PM, Andrew Purtell <[EMAIL PROTECTED]>wrote:
> Yeah, on HBase-0.94-security
>
> On Thu, Sep 13, 2012 at 1:34 PM, n keywal <[EMAIL PROTECTED]> wrote:
> > We've got two security builds on 0.94: HBase-0.94-security and
> > HBase-0.94-security-on-Hadoop-23
> > We can change the setting on one of them?
> >
> > On Thu, Sep 13, 2012 at 10:20 PM, Andrew Purtell <[EMAIL PROTECTED]
> >wrote:
> >
> >> Hey N,
> >>
> >> It makes sense to have a 0.94 build with -Psecurity enabled and local
> >> shortcut disabled. What do you think? For trunk builds your points sound
> >> good, and I have no opinion.
> >>
> >> On Thursday, September 13, 2012, n keywal wrote:
> >>
> >> > If I sum-up :
> >> >
> >> > We make it the default:
> >> > 1) Documented as such (what JD wrote already + a reference to
> HDFS-3907)
> >> > 2) Set in the config we use for the unit tests.
> >> >
> >> > Andrew, you said that we should keep some test with it off ? May be we
> >> > could do that in one of the existing build on the trunk, for example
> >> > HBase-TRUNK-on-Hadoop-2.0.0? i.e.: We run the tests on trunk with
> >> > short-circuit on; we run the test on hadoop-2 with short circuit off.
> >> >
> >> > I propose it this way because may be the tests will be faster with
> >> > short-circuit on, and as well if it's the default production config it
> >> > should be the one in the "main" test path.
> >> >
> >> > If that's ok, I can do test it on our test suite and do the jira for
> the
> >> > integration.
> >> >
> >> >
> >> >
> >> >
> >> > On Thu, Sep 13, 2012 at 8:31 PM, Andrew Purtell <[EMAIL PROTECTED]
> >> <javascript:;>>
> >> > wrote:
> >> >
> >> > > That's https://issues.apache.org/jira/browse/HBASE-4867
> >> > >
> >> > > On Thu, Sep 13, 2012 at 10:57 AM, Jesse Yates <
> [EMAIL PROTECTED]
> >> <javascript:;>
> >> > >
> >> > > wrote:
> >> > > > There was a jira a while back (can't remember the number) that I
> >> think
> >> > > came
> >> > > > from the Facebook fellas to have multiple hbase-site.xmls, with a
> >> > naming
> >> > > > scheme such that you can do layering of configurations.
> >> > > >
> >> > > > What ever happened to that?
> >> > > > -------------------
> >> > > > Jesse Yates
> >> > > > @jesse_yates
> >> > > > jyates.github.com
> >> > > >
> >> > > >
> >> > > > On Thu, Sep 13, 2012 at 10:44 AM, Andrew Purtell <
> >> [EMAIL PROTECTED]<javascript:;>
> >> > > >wrote:
> >> > > >
> >> > > >> Making it the default seems reasonable.
> >> > > >>
> >> > > >> For <= 0.94, the security profile has a separate hbase-site.xml
> file
> >> > > >> under test resources where there it could be defaulted as off.
> >> Reason
> >> > > >> is the local block sharing requires some shared group membership
> >> > > >> between HBase and HDFS service accounts that might be
> objectionable
> >> in
> >> > > >> a secure setting. So we should have test coverage with it off.
> >> > > >>
> >> > > >> On Thu, Sep 13, 2012 at 9:42 AM, n keywal <[EMAIL PROTECTED]
> >> <javascript:;>>
> >> > wrote:
> >> > > >> > It's all in the "pretty much" :-)
> >> > > >> > Should we make it the default (i.e. recommending it in the doc,
> >> > > emitting
> >> > > >> a
> >> > > >> > warning if it's not set) for all installation ?
> >> > > >> > Or would it make sense to separate the store and wal cases,
> >> meaning
> >> > > that
> >> > > >> we
> >> > > >> > need a change in hdfs to make this possible?
> >> > > >> >
> >> > > >> >
> >> > > >> > On Thu, Sep 13, 2012 at 6:30 PM, Jean-Daniel Cryans <
