Hi nifty devs,

After encountering HBASE-5702, I started playing with instant schema
updating (HBASE-4213) a bit more and I must say that it's a bit rough
which makes me wonder... should we pull that code out?

We're in this "interesting" situation in 0.94 where we have two
different ways to alter tables without disabling them and I don't
trust either. I'm pretty sure most of the devs don't even know which
one takes precedence over the other when both are enabled without
looking at the code. Well, right now hbase.online.schema.update.enable
needs to be enabled in order to have
hbase.instant.schema.alter.enabled working. If only the former is
enabled the master handles the alter, else if both are enabled then
it's going to be done via ZK although the master still keeps track of
it.

So the differences between both IIUC:

 - "Online schema update" is a rolling close/open of all the regions
so that they pick up the new HTD. It's handled by the master and has
been in since 0.92. I've used it quite a bit when running other tests
and it's ok as long as regions are not splitting and RS are not
shutting down. We also enabled it on our clusters here since our
regions don't tend to move that much.

 - "Instant schema alter" is instant in the sense that all the regions
are asked to close from the get-go but effectively the region servers
can only close one region at a time. The state of the alter is kept in
ZK and the master has a bunch of watches and logs the progress. It's
new since 0.94 and I'm not sure if anyone is using it. I've tested it
a bit and at the moment I can say the the MonitedTasks handling needs
to be redone, it logs way too much information in the log, but the few
alters I ran worked... It's just a bit hard to know when they're done.

FWIW we could pull either or both out, but instant schema alter hasn't
been in a released version yet so it's unlikely it'll bother someone
while the other is already in use (like here).

Opinions?

Thanks,

J-D

I agree having two implementations seems like a big mess. Any sense
which one is closer to working for the average user (i.e without
provisions like "don't split or crash servers mid-alter")?

On Mon, Apr 2, 2012 at 4:56 PM, Jean-Daniel Cryans <[EMAIL PROTECTED]> wrote:
> Hi nifty devs,
>
> After encountering HBASE-5702, I started playing with instant schema
> updating (HBASE-4213) a bit more and I must say that it's a bit rough
> which makes me wonder... should we pull that code out?
>
> We're in this "interesting" situation in 0.94 where we have two
> different ways to alter tables without disabling them and I don't
> trust either. I'm pretty sure most of the devs don't even know which
> one takes precedence over the other when both are enabled without
> looking at the code. Well, right now hbase.online.schema.update.enable
> needs to be enabled in order to have
> hbase.instant.schema.alter.enabled working. If only the former is
> enabled the master handles the alter, else if both are enabled then
> it's going to be done via ZK although the master still keeps track of
> it.
>
> So the differences between both IIUC:
>
>  - "Online schema update" is a rolling close/open of all the regions
> so that they pick up the new HTD. It's handled by the master and has
> been in since 0.92. I've used it quite a bit when running other tests
> and it's ok as long as regions are not splitting and RS are not
> shutting down. We also enabled it on our clusters here since our
> regions don't tend to move that much.
>
>  - "Instant schema alter" is instant in the sense that all the regions
> are asked to close from the get-go but effectively the region servers
> can only close one region at a time. The state of the alter is kept in
> ZK and the master has a bunch of watches and logs the progress. It's
> new since 0.94 and I'm not sure if anyone is using it. I've tested it
> a bit and at the moment I can say the the MonitedTasks handling needs
> to be redone, it logs way too much information in the log, but the few
> alters I ran worked... It's just a bit hard to know when they're done.
>
> FWIW we could pull either or both out, but instant schema alter hasn't
> been in a released version yet so it's unlikely it'll bother someone
> while the other is already in use (like here).
>
> Opinions?
>
> Thanks,
>
> J-D

--
Todd Lipcon
Software Engineer, Cloudera

On Mon, Apr 2, 2012 at 4:56 PM, Jean-Daniel Cryans <[EMAIL PROTECTED]> wrote:
> FWIW we could pull either or both out, but instant schema alter hasn't
> been in a released version yet so it's unlikely it'll bother someone
> while the other is already in use (like here).
>
> Opinions?
>

Lets not have two broke schema editors. Lets have just one broke
schema editor.  Would suggest back out instant schema alter and work
on hardening 'Online schema update' (though I dislike this one too..
its got its fingers in all over the codebase).

St.Ack

"instant schema alter" should endure the scenarios Todd mentioned.

I think we just need to simplify the usage of MonitoredTask there.

On Apr 2, 2012, at 4:58 PM, Todd Lipcon <[EMAIL PROTECTED]> wrote:

> I agree having two implementations seems like a big mess. Any sense
> which one is closer to working for the average user (i.e without
> provisions like "don't split or crash servers mid-alter")?
>
> On Mon, Apr 2, 2012 at 4:56 PM, Jean-Daniel Cryans <[EMAIL PROTECTED]> wrote:
>> Hi nifty devs,
>>
>> After encountering HBASE-5702, I started playing with instant schema
>> updating (HBASE-4213) a bit more and I must say that it's a bit rough
>> which makes me wonder... should we pull that code out?
>>
>> We're in this "interesting" situation in 0.94 where we have two
>> different ways to alter tables without disabling them and I don't
>> trust either. I'm pretty sure most of the devs don't even know which
>> one takes precedence over the other when both are enabled without
>> looking at the code. Well, right now hbase.online.schema.update.enable
>> needs to be enabled in order to have
>> hbase.instant.schema.alter.enabled working. If only the former is
>> enabled the master handles the alter, else if both are enabled then
>> it's going to be done via ZK although the master still keeps track of
>> it.
>>
>> So the differences between both IIUC:
>>
>>  - "Online schema update" is a rolling close/open of all the regions
>> so that they pick up the new HTD. It's handled by the master and has
>> been in since 0.92. I've used it quite a bit when running other tests
>> and it's ok as long as regions are not splitting and RS are not
>> shutting down. We also enabled it on our clusters here since our
>> regions don't tend to move that much.
>>
>>  - "Instant schema alter" is instant in the sense that all the regions
>> are asked to close from the get-go but effectively the region servers
>> can only close one region at a time. The state of the alter is kept in
>> ZK and the master has a bunch of watches and logs the progress. It's
>> new since 0.94 and I'm not sure if anyone is using it. I've tested it
>> a bit and at the moment I can say the the MonitedTasks handling needs
>> to be redone, it logs way too much information in the log, but the few
>> alters I ran worked... It's just a bit hard to know when they're done.
>>
>> FWIW we could pull either or both out, but instant schema alter hasn't
>> been in a released version yet so it's unlikely it'll bother someone
>> while the other is already in use (like here).
>>
>> Opinions?
>>
>> Thanks,
>>
>> J-D
>
>
>
> --
> Todd Lipcon
> Software Engineer, Cloudera

Agree about pulling out something that is broken.

However, it should at least be possible to update
via the shell parameters such as split threshold with instantaneous
effect and no region offlining. I've encountered situations where it is really necessary to be able to alter the attributes of existing tables without the "disruptions in the force" of a rolling cluster configuration update. Things like adjusting the split threshold of a table growing a lot larger than anticipated (while obviously given the write rate region availability is important).
I don't have an opinion about addition or removal of column families, but online CF addition could be useful when evolving application design yet maintaining high availability.
Best regards,
    - Andy
Problems worthy of attack prove their worth by hitting back. - Piet Hein (via Tom White)

>________________________________
> From: Jean-Daniel Cryans <[EMAIL PROTECTED]>
>To: [EMAIL PROTECTED]
>Sent: Monday, April 2, 2012 4:56 PM
>Subject: Pull instant schema updating out?
>
>Hi nifty devs,
>
>After encountering HBASE-5702, I started playing with instant schema
>updating (HBASE-4213) a bit more and I must say that it's a bit rough
>which makes me wonder... should we pull that code out?
>
>We're in this "interesting" situation in 0.94 where we have two
>different ways to alter tables without disabling them and I don't
>trust either. I'm pretty sure most of the devs don't even know which
>one takes precedence over the other when both are enabled without
>looking at the code. Well, right now hbase.online.schema.update.enable
>needs to be enabled in order to have
>hbase.instant.schema.alter.enabled working. If only the former is
>enabled the master handles the alter, else if both are enabled then
>it's going to be done via ZK although the master still keeps track of
>it.
>
>So the differences between both IIUC:
>
>- "Online schema update" is a rolling close/open of all the regions
>so that they pick up the new HTD. It's handled by the master and has
>been in since 0.92. I've used it quite a bit when running other tests
>and it's ok as long as regions are not splitting and RS are not
>shutting down. We also enabled it on our clusters here since our
>regions don't tend to move that much.
>
>- "Instant schema alter" is instant in the sense that all the regions
>are asked to close from the get-go but effectively the region servers
>can only close one region at a time. The state of the alter is kept in
>ZK and the master has a bunch of watches and logs the progress. It's
>new since 0.94 and I'm not sure if anyone is using it. I've tested it
>a bit and at the moment I can say the the MonitedTasks handling needs
>to be redone, it logs way too much information in the log, but the few
>alters I ran worked... It's just a bit hard to know when they're done.
>
>FWIW we could pull either or both out, but instant schema alter hasn't
>been in a released version yet so it's unlikely it'll bother someone
>while the other is already in use (like here).
>
>Opinions?
>
>Thanks,
>
>J-D
>
>
>

How much duplicate code are we talking about? Do these share common code to close/open the regions, or is it a completely different code path? (I know I could look myself, but it's easier to ask)

From a gut reaction it feels cleaner to keep track of the state of the alter in ZK.

________________________________
 From: Jean-Daniel Cryans <[EMAIL PROTECTED]>
To: [EMAIL PROTECTED]
Sent: Monday, April 2, 2012 4:56 PM
Subject: Pull instant schema updating out?
 
Hi nifty devs,

After encountering HBASE-5702, I started playing with instant schema
updating (HBASE-4213) a bit more and I must say that it's a bit rough
which makes me wonder... should we pull that code out?

We're in this "interesting" situation in 0.94 where we have two
different ways to alter tables without disabling them and I don't
trust either. I'm pretty sure most of the devs don't even know which
one takes precedence over the other when both are enabled without
looking at the code. Well, right now hbase.online.schema.update.enable
needs to be enabled in order to have
hbase.instant.schema.alter.enabled working. If only the former is
enabled the master handles the alter, else if both are enabled then
it's going to be done via ZK although the master still keeps track of
it.

So the differences between both IIUC:

- "Online schema update" is a rolling close/open of all the regions
so that they pick up the new HTD. It's handled by the master and has
been in since 0.92. I've used it quite a bit when running other tests
and it's ok as long as regions are not splitting and RS are not
shutting down. We also enabled it on our clusters here since our
regions don't tend to move that much.

- "Instant schema alter" is instant in the sense that all the regions
are asked to close from the get-go but effectively the region servers
can only close one region at a time. The state of the alter is kept in
ZK and the master has a bunch of watches and logs the progress. It's
new since 0.94 and I'm not sure if anyone is using it. I've tested it
a bit and at the moment I can say the the MonitedTasks handling needs
to be redone, it logs way too much information in the log, but the few
alters I ran worked... It's just a bit hard to know when they're done.

FWIW we could pull either or both out, but instant schema alter hasn't
been in a released version yet so it's unlikely it'll bother someone
while the other is already in use (like here).

Opinions?

Thanks,

J-D

The two approaches are quite different.
See related discussion here:
http://search-hadoop.com/m/pC2q8ke5nr/HBASE-1730+and+HBASE-4213&subj=Re+HBASE+1730+and+HBASE+4213

On Mon, Apr 2, 2012 at 6:04 PM, lars hofhansl <[EMAIL PROTECTED]> wrote:

> How much duplicate code are we talking about? Do these share common code
> to close/open the regions, or is it a completely different code path? (I
> know I could look myself, but it's easier to ask)
>
> From a gut reaction it feels cleaner to keep track of the state of the
> alter in ZK.
>
>
>
> ________________________________
>  From: Jean-Daniel Cryans <[EMAIL PROTECTED]>
> To: [EMAIL PROTECTED]
> Sent: Monday, April 2, 2012 4:56 PM
> Subject: Pull instant schema updating out?
>
> Hi nifty devs,
>
> After encountering HBASE-5702, I started playing with instant schema
> updating (HBASE-4213) a bit more and I must say that it's a bit rough
> which makes me wonder... should we pull that code out?
>
> We're in this "interesting" situation in 0.94 where we have two
> different ways to alter tables without disabling them and I don't
> trust either. I'm pretty sure most of the devs don't even know which
> one takes precedence over the other when both are enabled without
> looking at the code. Well, right now hbase.online.schema.update.enable
> needs to be enabled in order to have
> hbase.instant.schema.alter.enabled working. If only the former is
> enabled the master handles the alter, else if both are enabled then
> it's going to be done via ZK although the master still keeps track of
> it.
>
> So the differences between both IIUC:
>
> - "Online schema update" is a rolling close/open of all the regions
> so that they pick up the new HTD. It's handled by the master and has
> been in since 0.92. I've used it quite a bit when running other tests
> and it's ok as long as regions are not splitting and RS are not
> shutting down. We also enabled it on our clusters here since our
> regions don't tend to move that much.
>
> - "Instant schema alter" is instant in the sense that all the regions
> are asked to close from the get-go but effectively the region servers
> can only close one region at a time. The state of the alter is kept in
> ZK and the master has a bunch of watches and logs the progress. It's
> new since 0.94 and I'm not sure if anyone is using it. I've tested it
> a bit and at the moment I can say the the MonitedTasks handling needs
> to be redone, it logs way too much information in the log, but the few
> alters I ran worked... It's just a bit hard to know when they're done.
>
> FWIW we could pull either or both out, but instant schema alter hasn't
> been in a released version yet so it's unlikely it'll bother someone
> while the other is already in use (like here).
>
> Opinions?
>
> Thanks,
>
> J-D
>

On Mon, Apr 2, 2012 at 6:04 PM, lars hofhansl <[EMAIL PROTECTED]> wrote:
> How much duplicate code are we talking about? Do these share common code to close/open the regions, or is it a completely different code path? (I know I could look myself, but it's easier to ask)
>
> From a gut reaction it feels cleaner to keep track of the state of the alter in ZK.
>
>

Both are incomplete.  The approaches share no code.  The conversation
Ted cites is from nearly a year ago (and is nought but our platitudes
that the authors get together on the problem which didn't happen; the
context did not help w/ one part-time on the job and the other a
summer intern).  Neither has a sponsor.  The "Online schema update" is
at least used in a few places, cautiously.  What should happen is we
should pull out both and replace with a working online schema
mechanism.  Until that happens undo at least one of them before
someone other than J-D gets hurt.  I vote for pulling  "Online schema
update".

St.Ack

>> I vote for pulling  "Online schema update".
I have the same vote.

On Mon, Apr 2, 2012 at 9:05 PM, Stack <[EMAIL PROTECTED]> wrote:

> On Mon, Apr 2, 2012 at 6:04 PM, lars hofhansl <[EMAIL PROTECTED]> wrote:
> > How much duplicate code are we talking about? Do these share common code
> to close/open the regions, or is it a completely different code path? (I
> know I could look myself, but it's easier to ask)
> >
> > From a gut reaction it feels cleaner to keep track of the state of the
> alter in ZK.
> >
> >
>
> Both are incomplete.  The approaches share no code.  The conversation
> Ted cites is from nearly a year ago (and is nought but our platitudes
> that the authors get together on the problem which didn't happen; the
> context did not help w/ one part-time on the job and the other a
> summer intern).  Neither has a sponsor.  The "Online schema update" is
> at least used in a few places, cautiously.  What should happen is we
> should pull out both and replace with a working online schema
> mechanism.  Until that happens undo at least one of them before
> someone other than J-D gets hurt.  I vote for pulling  "Online schema
> update".
>
> St.Ack
>

On Mon, Apr 2, 2012 at 9:05 PM, Stack <[EMAIL PROTECTED]> wrote:
> I vote for pulling  "Online schema  update".
>

Sorry, I meant "Instant schema alter", the one new to 0.94.  "Online
schema  update" shipped in 0.92 and is used in a few places in
production.
St.Ack

I did a superficial review of HBASE-4213 and remember I was quite exited about the promise of the change at
that time. If the problem is just with the MonitoredTask behavior (as indicated in HBASE-5702) I'd say we try to fix that.

-- Lars

________________________________
 From: Stack <[EMAIL PROTECTED]>
To: [EMAIL PROTECTED]; lars hofhansl <[EMAIL PROTECTED]>
Sent: Monday, April 2, 2012 9:05 PM
Subject: Re: Pull instant schema updating out?
 
On Mon, Apr 2, 2012 at 6:04 PM, lars hofhansl <[EMAIL PROTECTED]> wrote:
> How much duplicate code are we talking about? Do these share common code to close/open the regions, or is it a completely different code path? (I know I could look myself, but it's easier to ask)
>
> From a gut reaction it feels cleaner to keep track of the state of the alter in ZK.
>
>

Both are incomplete.  The approaches share no code.  The conversation
Ted cites is from nearly a year ago (and is nought but our platitudes
that the authors get together on the problem which didn't happen; the
context did not help w/ one part-time on the job and the other a
summer intern).  Neither has a sponsor.  The "Online schema update" is
at least used in a few places, cautiously.  What should happen is we
should pull out both and replace with a working online schema
mechanism.  Until that happens undo at least one of them before
someone other than J-D gets hurt.  I vote for pulling  "Online schema
update".

St.Ack

I agree with Lars' assessment.

On Apr 2, 2012, at 11:07 PM, lars hofhansl <[EMAIL PROTECTED]> wrote:

> I did a superficial review of HBASE-4213 and remember I was quite exited about the promise of the change at
> that time. If the problem is just with the MonitoredTask behavior (as indicated in HBASE-5702) I'd say we try to fix that.
>
> -- Lars
>
>
>
> ________________________________
> From: Stack <[EMAIL PROTECTED]>
> To: [EMAIL PROTECTED]; lars hofhansl <[EMAIL PROTECTED]>
> Sent: Monday, April 2, 2012 9:05 PM
> Subject: Re: Pull instant schema updating out?
>
> On Mon, Apr 2, 2012 at 6:04 PM, lars hofhansl <[EMAIL PROTECTED]> wrote:
>> How much duplicate code are we talking about? Do these share common code to close/open the regions, or is it a completely different code path? (I know I could look myself, but it's easier to ask)
>>
>> From a gut reaction it feels cleaner to keep track of the state of the alter in ZK.
>>
>>
>
> Both are incomplete.  The approaches share no code.  The conversation
> Ted cites is from nearly a year ago (and is nought but our platitudes
> that the authors get together on the problem which didn't happen; the
> context did not help w/ one part-time on the job and the other a
> summer intern).  Neither has a sponsor.  The "Online schema update" is
> at least used in a few places, cautiously.  What should happen is we
> should pull out both and replace with a working online schema
> mechanism.  Until that happens undo at least one of them before
> someone other than J-D gets hurt.  I vote for pulling  "Online schema
> update".
>
> St.Ack

On Mon, Apr 2, 2012 at 11:07 PM, lars hofhansl <[EMAIL PROTECTED]> wrote:
> If the problem is just with the MonitoredTask behavior (as indicated in HBASE-5702) I'd say we try to fix that.
>

Who is the 'we' in the above?  Its not been touched since committed.
Other issues are it does an explosive close-all and then open-all on a
table; there should be dampening.  How about pulling it out till its
fixed.
St.Ack

What about moving one or both implementations into a feature branch?
If there's demand, someone will maintain it there and when it looks
good enough for general use, we merge it back in?

-Joey

On Tue, Apr 3, 2012 at 2:50 PM, Stack <[EMAIL PROTECTED]> wrote:
> On Mon, Apr 2, 2012 at 11:07 PM, lars hofhansl <[EMAIL PROTECTED]> wrote:
>> If the problem is just with the MonitoredTask behavior (as indicated in HBASE-5702) I'd say we try to fix that.
>>
>
> Who is the 'we' in the above?  Its not been touched since committed.
> Other issues are it does an explosive close-all and then open-all on a
> table; there should be dampening.  How about pulling it out till its
> fixed.
> St.Ack

--
Joey Echeverria
Senior Solutions Architect
Cloudera, Inc.

If we pull it out it'll never get fixed. :)

I don't really have strong feelings about this, just that functional schema changes without disabling the table are a pretty important feature.
If it's not working and there's no reasonable chance to get it fixed, because nobody is working on it, then let's pull it.

________________________________
 From: Stack <[EMAIL PROTECTED]>
To: [EMAIL PROTECTED]; lars hofhansl <[EMAIL PROTECTED]>
Sent: Tuesday, April 3, 2012 7:50 AM
Subject: Re: Pull instant schema updating out?
 
On Mon, Apr 2, 2012 at 11:07 PM, lars hofhansl <[EMAIL PROTECTED]> wrote:
> If the problem is just with the MonitoredTask behavior (as indicated in HBASE-5702) I'd say we try to fix that.
>

Who is the 'we' in the above?  Its not been touched since committed.
Other issues are it does an explosive close-all and then open-all on a
table; there should be dampening.  How about pulling it out till its
fixed.
St.Ack

Has any change been done in the past half year on "Online schema update" ?

When we make such an important decision, we should evaluate various
factors. "Instant schema alter" has better design: it endures master
failover.

If the following tests can be written for "Online schema update" and pass,
I would vote for it:
src/test/java/org/apache/hadoop/hbase/client/TestInstantSchemaChange.java
src/test/java/org/apache/hadoop/hbase/client/TestInstantSchemaChangeFailover.java
src/test/java/org/apache/hadoop/hbase/client/TestInstantSchemaChangeSplit.java

Otherwise I am -1 on pulling "Instant schema alter"

There is more work to be done: reduce the noise posed by MonitoredTask,
throttling, etc. But these are not difficult tasks.

On Tue, Apr 3, 2012 at 9:14 AM, lars hofhansl <[EMAIL PROTECTED]> wrote:

> If we pull it out it'll never get fixed. :)
>
> I don't really have strong feelings about this, just that functional
> schema changes without disabling the table are a pretty important feature.
> If it's not working and there's no reasonable chance to get it fixed,
> because nobody is working on it, then let's pull it.
>
>
>
> ________________________________
>  From: Stack <[EMAIL PROTECTED]>
> To: [EMAIL PROTECTED]; lars hofhansl <[EMAIL PROTECTED]>
> Sent: Tuesday, April 3, 2012 7:50 AM
> Subject: Re: Pull instant schema updating out?
>
> On Mon, Apr 2, 2012 at 11:07 PM, lars hofhansl <[EMAIL PROTECTED]>
> wrote:
> > If the problem is just with the MonitoredTask behavior (as indicated in
> HBASE-5702) I'd say we try to fix that.
> >
>
> Who is the 'we' in the above?  Its not been touched since committed.
> Other issues are it does an explosive close-all and then open-all on a
> table; there should be dampening.  How about pulling it out till its
> fixed.
> St.Ack
>

On Tue, Apr 3, 2012 at 9:14 AM, lars hofhansl <[EMAIL PROTECTED]> wrote:
> If we pull it out it'll never get fixed. :)
>

Going by recent past, leaving it in doesn't get it fixed either (Its
not even used it seems else someone would have complained before J-Ds
mail yesterday).

> I don't really have strong feelings about this, just that functional schema changes without disabling the table are a pretty important feature.
> If it's not working and there's no reasonable chance to get it fixed, because nobody is working on it, then let's pull it.
>

Agree.  Could pull it and put it on a branch as Joey suggests.
St.Ack

I found this problem after looking at it for 5 minutes and it features
bugs without even being enabled. I'm wondering if that tree is hiding
a massive forrest or not.

J-D

On Mon, Apr 2, 2012 at 11:07 PM, lars hofhansl <[EMAIL PROTECTED]> wrote:
> I did a superficial review of HBASE-4213 and remember I was quite exited about the promise of the change at
> that time. If the problem is just with the MonitoredTask behavior (as indicated in HBASE-5702) I'd say we try to fix that.
>
> -- Lars
>
>
>
> ________________________________
>  From: Stack <[EMAIL PROTECTED]>
> To: [EMAIL PROTECTED]; lars hofhansl <[EMAIL PROTECTED]>
> Sent: Monday, April 2, 2012 9:05 PM
> Subject: Re: Pull instant schema updating out?
>
> On Mon, Apr 2, 2012 at 6:04 PM, lars hofhansl <[EMAIL PROTECTED]> wrote:
>> How much duplicate code are we talking about? Do these share common code to close/open the regions, or is it a completely different code path? (I know I could look myself, but it's easier to ask)
>>
>> From a gut reaction it feels cleaner to keep track of the state of the alter in ZK.
>>
>>
>
> Both are incomplete.  The approaches share no code.  The conversation
> Ted cites is from nearly a year ago (and is nought but our platitudes
> that the authors get together on the problem which didn't happen; the
> context did not help w/ one part-time on the job and the other a
> summer intern).  Neither has a sponsor.  The "Online schema update" is
> at least used in a few places, cautiously.  What should happen is we
> should pull out both and replace with a working online schema
> mechanism.  Until that happens undo at least one of them before
> someone other than J-D gets hurt.  I vote for pulling  "Online schema
> update".
>
> St.Ack

On Tue, Apr 3, 2012 at 9:23 AM, Ted Yu <[EMAIL PROTECTED]> wrote:
> Has any change been done in the past half year on "Online schema update" ?
>

Minor if any.

It does have the advantage that it is at least being used.
> When we make such an important decision, we should evaluate various
> factors. "Instant schema alter" has better design: it endures master
> failover.
>
> If the following tests can be written for "Online schema update" and pass,
> I would vote for it:
> src/test/java/org/apache/hadoop/hbase/client/TestInstantSchemaChange.java
> src/test/java/org/apache/hadoop/hbase/client/TestInstantSchemaChangeFailover.java
> src/test/java/org/apache/hadoop/hbase/client/TestInstantSchemaChangeSplit.java
>
> Otherwise I am -1 on pulling "Instant schema alter"
>
> There is more work to be done: reduce the noise posed by MonitoredTask,
> throttling, etc. But these are not difficult tasks.
>

It has 'better' design but it is fallible to failure still and far
from perfect in that it persists by writing feature-particular znodes
w/ feature-particular custom handlers sprinkled about the code base.
We should be able to do better.

All of your above argument is posited on more work being done whether
tests for one implementation or completion of a feature no one has
used in almost a year.   I would suggest that you don't have much of a
case if it is predicated on the work of others.  Sign up to fix it or
lets pull it; even then I'd say pull it till its fixed rather than let
a broke implementation go out in 0.94.

St.Ack

> How about pulling it out till its fixed.

+1
More than one committer has raised concerns about the code. It seems pretty clear what to do in this case. Revert, then proceed.
 
Best regards,
    - Andy
Problems worthy of attack prove their worth by hitting back. - Piet Hein (via Tom White)

>________________________________
> From: Stack <[EMAIL PROTECTED]>
>To: [EMAIL PROTECTED]; lars hofhansl <[EMAIL PROTECTED]>
>Sent: Tuesday, April 3, 2012 7:50 AM
>Subject: Re: Pull instant schema updating out?
>
>On Mon, Apr 2, 2012 at 11:07 PM, lars hofhansl <[EMAIL PROTECTED]> wrote:
>> If the problem is just with the MonitoredTask behavior (as indicated in HBASE-5702) I'd say we try to fix that.
>>
>
>Who is the 'we' in the above?  Its not been touched since committed.
>Other issues are it does an explosive close-all and then open-all on a
>table; there should be dampening.  How about pulling it out till its
>fixed.
>St.Ack
>
>
>

We're using a variant of the Online schema update in our 89 production.
There are significant differences because of the master rewrite, so I
can't speak about the stability on trunk.  We don't run with online
splitting, so that's also a large variant from the use cases some of you
are trying to support.  Overall, I'll echo my original thoughts when
HBASE-1730 was submitted that I think have played out: Online Schema
Changes have looser requirements that allow us to create a solid solution
with less effort.  Optimally, production schema changes shouldn't happen
very often and will be actively monitored when they do.  Master failure
just means that you need to retry the operation, not that data loss or
region unavailability occurred.  Unless this is a critical long-term
feature for your production environment, the persistence is overly
elaborate and will cause more problems with downtime/analysis/support than
it will solve with the minor inconvenience of checking back in 5 min to
make sure your change completely rolled out.  I think there are a million
other features of higher benefit and lower support effort.

I think we should stay at 1 version, Online Schema Update.  It's the
simplest and most tested.  If we go with the more complicated "Instant
Schema Alter", I should be owned by a group of full time devs who think
it's the one of the flagship features of their production system, because
it's a major undertaking and deserves that amount of effort.

On 4/3/12 12:49 PM, "Stack" <[EMAIL PROTECTED]> wrote:

>On Tue, Apr 3, 2012 at 9:23 AM, Ted Yu <[EMAIL PROTECTED]> wrote:
>> Has any change been done in the past half year on "Online schema
>>update" ?
>>
>
>Minor if any.
>
>It does have the advantage that it is at least being used.
>
>
>> When we make such an important decision, we should evaluate various
>> factors. "Instant schema alter" has better design: it endures master
>> failover.
>>
>> If the following tests can be written for "Online schema update" and
>>pass,
>> I would vote for it:
>>
>>src/test/java/org/apache/hadoop/hbase/client/TestInstantSchemaChange.java
>>
>>src/test/java/org/apache/hadoop/hbase/client/TestInstantSchemaChangeFailo
>>ver.java
>>
>>src/test/java/org/apache/hadoop/hbase/client/TestInstantSchemaChangeSplit
>>.java
>>
>> Otherwise I am -1 on pulling "Instant schema alter"
>>
>> There is more work to be done: reduce the noise posed by MonitoredTask,
>> throttling, etc. But these are not difficult tasks.
>>
>
>It has 'better' design but it is fallible to failure still and far
>from perfect in that it persists by writing feature-particular znodes
>w/ feature-particular custom handlers sprinkled about the code base.
>We should be able to do better.
>
>All of your above argument is posited on more work being done whether
>tests for one implementation or completion of a feature no one has
>used in almost a year.   I would suggest that you don't have much of a
>case if it is predicated on the work of others.  Sign up to fix it or
>lets pull it; even then I'd say pull it till its fixed rather than let
>a broke implementation go out in 0.94.
>
>St.Ack

On Tue, Apr 3, 2012 at 10:34 AM, Nicolas Spiegelberg
<[EMAIL PROTECTED]> wrote:
> I think we should stay at 1 version, Online Schema Update.  It's the
> simplest and most tested.  If we go with the more complicated "Instant
> Schema Alter", I should be owned by a group of full time devs who think
> it's the one of the flagship features of their production system, because
> it's a major undertaking and deserves that amount of effort.
>

I made HBASE-5715.  Will do it before 0.94.0rc1.
St.Ack

Subbu has signed on HBASE-5702 and created HBASE-5713: Introduce throttling
during Instant schema change process to throttle opening/closing regions.

What's the name of the branch that would carry "Instant Schema Alter" ?

Cheers

On Wed, Apr 4, 2012 at 9:44 AM, Stack <[EMAIL PROTECTED]> wrote:

> On Tue, Apr 3, 2012 at 10:34 AM, Nicolas Spiegelberg
> <[EMAIL PROTECTED]> wrote:
> > I think we should stay at 1 version, Online Schema Update.  It's the
> > simplest and most tested.  If we go with the more complicated "Instant
> > Schema Alter", I should be owned by a group of full time devs who think
> > it's the one of the flagship features of their production system, because
> > it's a major undertaking and deserves that amount of effort.
> >
>
> I made HBASE-5715.  Will do it before 0.94.0rc1.
> St.Ack
>