Hi guys,
 
Mid next year, we agreed on a release process documented in this thread: http://www.mail-archive.com/[EMAIL PROTECTED]/msg04172.html.
 
Since then, we have not really followed either of its two rules:
 
(1) Frequent (every 3 month releases)
(2) Branch stability (only P1 issues on the branch).
 
So I wanted to revisit our release procedure to make sure we have one that we can actually follow.
 
For us at Yahoo, branch stability is very important since we release all the patches directly from the branch. If we can't rely on the fact that only critical fixes go in, we will need to resort to git branches that will make the whole process very comberson because we now need to hand pick patches from the apache branch and port them onto our private branch. I would imaging that others using Pig in production would have similar issues.
 
Olga
 
 
Olga

I am all for maintaining stability of branches, and the trunk, as everyone benefits from it.  But I do not think this means we should limit bug fixing in the branches to only critical issues.  As Pig gets more users we have more and more people on older branches who will want fixes for bugs without dealing with bigger version changes.  So I am not in favor of limiting checkins to branches to P1 issues.

What if we maintain stability on the branches by quickly reverting any patches that break the build, the unit tests, or the e2e tests?  This allows us to move forward with bug fix versions, it allows those who depend on branch stability (which I suspect is everyone in the distribution business plus everyone rolling their own Pig), and it should promote developer responsibility (no one likes having their patches reverted).

Alan.
  
On Nov 2, 2012, at 3:58 PM, Olga Natkovich wrote:

> Hi guys,
>  
> Mid next year, we agreed on a release process documented in this thread: http://www.mail-archive.com/[EMAIL PROTECTED]/msg04172.html.
>  
> Since then, we have not really followed either of its two rules:
>  
> (1) Frequent (every 3 month releases)
> (2) Branch stability (only P1 issues on the branch).
>  
> So I wanted to revisit our release procedure to make sure we have one that we can actually follow.
>  
> For us at Yahoo, branch stability is very important since we release all the patches directly from the branch. If we can't rely on the fact that only critical fixes go in, we will need to resort to git branches that will make the whole process very comberson because we now need to hand pick patches from the apache branch and port them onto our private branch. I would imaging that others using Pig in production would have similar issues.
>  
> Olga
>  
>  
> Olga

Hi,

I agree with what Alan says and I like his proposal.
However, to make it feasible, we need to make jenkins builds stable,
otherwise a real problem introduced by a patch might be lost in the
hundreds of failures due to clover licenses, minicluster issues, etc...

I don't like too much making jenkins post to jira the results of the build
after a patch is committed, as it pollutes the jira itself, however in this
case it might be a good way to promote developer responsibility. Is it
possible to activate this only for branches?

Cheers,
--
Gianmarco

On Fri, Nov 2, 2012 at 8:19 PM, Alan Gates <[EMAIL PROTECTED]> wrote:

> I am all for maintaining stability of branches, and the trunk, as everyone
> benefits from it.  But I do not think this means we should limit bug fixing
> in the branches to only critical issues.  As Pig gets more users we have
> more and more people on older branches who will want fixes for bugs without
> dealing with bigger version changes.  So I am not in favor of limiting
> checkins to branches to P1 issues.
>
> What if we maintain stability on the branches by quickly reverting any
> patches that break the build, the unit tests, or the e2e tests?  This
> allows us to move forward with bug fix versions, it allows those who depend
> on branch stability (which I suspect is everyone in the distribution
> business plus everyone rolling their own Pig), and it should promote
> developer responsibility (no one likes having their patches reverted).
>
> Alan.
>
> On Nov 2, 2012, at 3:58 PM, Olga Natkovich wrote:
>
> > Hi guys,
> >
> > Mid next year, we agreed on a release process documented in this thread:
> http://www.mail-archive.com/[EMAIL PROTECTED]/msg04172.html.
> >
> > Since then, we have not really followed either of its two rules:
> >
> > (1) Frequent (every 3 month releases)
> > (2) Branch stability (only P1 issues on the branch).
> >
> > So I wanted to revisit our release procedure to make sure we have one
> that we can actually follow.
> >
> > For us at Yahoo, branch stability is very important since we release all
> the patches directly from the branch. If we can't rely on the fact that
> only critical fixes go in, we will need to resort to git branches that will
> make the whole process very comberson because we now need to hand pick
> patches from the apache branch and port them onto our private branch. I
> would imaging that others using Pig in production would have similar issues.
> >
> > Olga
> >
> >
> > Olga
>
>

I can see how this would work for research projects but for real production this will not work. And I actually meant much more stringent stability. I don't think we should commit patches to either trunk or branch that destabilize the tree. We used to run full regression before each commit - is this no longer the case? By stability I meant very few things go into the branch. I know that pig has pretty decent tests - better coverage than many other projects. However, we do not have any testing at scale and inevitably, users end up doing testing. So any time we deploy new major version, it takes us at least a month to get it stable and once it is stabilized we want to keep it this way.

So for us at Yahoo, the only way to work directly from the branch is to go by our original plan. If that is not possible, we would go with the private git branch.

Olga
________________________________
 From: Alan Gates <[EMAIL PROTECTED]>
To: [EMAIL PROTECTED]
Sent: Friday, November 2, 2012 8:19 PM
Subject: Re: Our release process
 
I am all for maintaining stability of branches, and the trunk, as everyone benefits from it.  But I do not think this means we should limit bug fixing in the branches to only critical issues.  As Pig gets more users we have more and more people on older branches who will want fixes for bugs without dealing with bigger version changes.  So I am not in favor of limiting checkins to branches to P1 issues.

What if we maintain stability on the branches by quickly reverting any patches that break the build, the unit tests, or the e2e tests?  This allows us to move forward with bug fix versions, it allows those who depend on branch stability (which I suspect is everyone in the distribution business plus everyone rolling their own Pig), and it should promote developer responsibility (no one likes having their patches reverted).

Alan.
 
On Nov 2, 2012, at 3:58 PM, Olga Natkovich wrote:

> Hi guys,
> 
> Mid next year, we agreed on a release process documented in this thread: http://www.mail-archive.com/[EMAIL PROTECTED]/msg04172.html.
> 
> Since then, we have not really followed either of its two rules:
> 
> (1) Frequent (every 3 month releases)
> (2) Branch stability (only P1 issues on the branch).
> 
> So I wanted to revisit our release procedure to make sure we have one that we can actually follow.
> 
> For us at Yahoo, branch stability is very important since we release all the patches directly from the branch. If we can't rely on the fact that only critical fixes go in, we will need to resort to git branches that will make the whole process very comberson because we now need to hand pick patches from the apache branch and port them onto our private branch. I would imaging that others using Pig in production would have similar issues.
> 
> Olga
> 
> 
> Olga

Hi,

Sure we don't want to commit patches that destabilize the code base.
However, unfortunately, there is no way to know whether a patch will
destabilize the code or not. Even testing is only a heuristic. So how do we
draw the line?
We seem to agree that only bug fixing should go into branches. However it
seems that we have two different views on the policy: Olga is proposing to
have only P1 bugs fixed, while Alan is suggesting to be more lax on what
goes into the branches.
Regardless of the policy chosen, how do we define the priority of a bug? By
how many users are affected? By whether it can corrupt data? Is there a
formal definition we can agree on? Otherwise defining a policy becomes hard.

The test-commit task does not run full regression because the full test
suite takes too long to execute. And I agree that asking to run the full
test suite before committing any change slows down the (already slow)
review process.
However, I would be fine with running the full test suite for bug fixes
that need to go into branches, in order to guarantee absence of regressions.

Cheers,
--
Gianmarco

On Sun, Nov 4, 2012 at 5:17 PM, Olga Natkovich <[EMAIL PROTECTED]> wrote:

> I can see how this would work for research projects but for real
> production this will not work. And I actually meant much more stringent
> stability. I don't think we should commit patches to either trunk or branch
> that destabilize the tree. We used to run full regression before each
> commit - is this no longer the case? By stability I meant very few things
> go into the branch. I know that pig has pretty decent tests - better
> coverage than many other projects. However, we do not have any testing at
> scale and inevitably, users end up doing testing. So any time we deploy new
> major version, it takes us at least a month to get it stable and once it is
> stabilized we want to keep it this way.
>
> So for us at Yahoo, the only way to work directly from the branch is to go
> by our original plan. If that is not possible, we would go with the private
> git branch.
>
> Olga
>
>
> ________________________________
>  From: Alan Gates <[EMAIL PROTECTED]>
> To: [EMAIL PROTECTED]
> Sent: Friday, November 2, 2012 8:19 PM
> Subject: Re: Our release process
>
> I am all for maintaining stability of branches, and the trunk, as everyone
> benefits from it.  But I do not think this means we should limit bug fixing
> in the branches to only critical issues.  As Pig gets more users we have
> more and more people on older branches who will want fixes for bugs without
> dealing with bigger version changes.  So I am not in favor of limiting
> checkins to branches to P1 issues.
>
> What if we maintain stability on the branches by quickly reverting any
> patches that break the build, the unit tests, or the e2e tests?  This
> allows us to move forward with bug fix versions, it allows those who depend
> on branch stability (which I suspect is everyone in the distribution
> business plus everyone rolling their own Pig), and it should promote
> developer responsibility (no one likes having their patches reverted).
>
> Alan.
>
> On Nov 2, 2012, at 3:58 PM, Olga Natkovich wrote:
>
> > Hi guys,
> >
> > Mid next year, we agreed on a release process documented in this thread:
> http://www.mail-archive.com/[EMAIL PROTECTED]/msg04172.html.
> >
> > Since then, we have not really followed either of its two rules:
> >
> > (1) Frequent (every 3 month releases)
> > (2) Branch stability (only P1 issues on the branch).
> >
> > So I wanted to revisit our release procedure to make sure we have one
> that we can actually follow.
> >
> > For us at Yahoo, branch stability is very important since we release all
> the patches directly from the branch. If we can't rely on the fact that
> only critical fixes go in, we will need to resort to git branches that will
> make the whole process very comberson because we now need to hand pick
> patches from the apache branch and port them onto our private branch. I

Hi Gianmarco,

Thanks for your comments. Here is a little more information.

At Yahoo, we consider the following issues to be P1:

(1) Bugs that cause wrong results being produced silently
(2) Bugs that cause failures with no easy workaround

Regarding tests. I would suggest we have different rules for trunk and branches:

(1) For branches, I think we should run the full regression suite (including e2e) prior to commit. This way we can ensure branch stability and, as number of patches should be small, will not be a burden
(2) For trunk, we can go with test-commit only and fix things quickly when things break.

Olga
________________________________
From: Gianmarco De Francisci Morales <[EMAIL PROTECTED]>
To: [EMAIL PROTECTED]; Olga Natkovich <[EMAIL PROTECTED]>
Sent: Monday, November 5, 2012 10:37 AM
Subject: Re: Our release process

Hi,

Sure we don't want to commit patches that destabilize the code base.
However, unfortunately, there is no way to know whether a patch will
destabilize the code or not. Even testing is only a heuristic. So how do we
draw the line?
We seem to agree that only bug fixing should go into branches. However it
seems that we have two different views on the policy: Olga is proposing to
have only P1 bugs fixed, while Alan is suggesting to be more lax on what
goes into the branches.
Regardless of the policy chosen, how do we define the priority of a bug? By
how many users are affected? By whether it can corrupt data? Is there a
formal definition we can agree on? Otherwise defining a policy becomes hard.

The test-commit task does not run full regression because the full test
suite takes too long to execute. And I agree that asking to run the full
test suite before committing any change slows down the (already slow)
review process.
However, I would be fine with running the full test suite for bug fixes
that need to go into branches, in order to guarantee absence of regressions.

Cheers,
--
Gianmarco

On Sun, Nov 4, 2012 at 5:17 PM, Olga Natkovich <[EMAIL PROTECTED]> wrote:

> I can see how this would work for research projects but for real
> production this will not work. And I actually meant much more stringent
> stability. I don't think we should commit patches to either trunk or branch
> that destabilize the tree. We used to run full regression before each
> commit - is this no longer the case? By stability I meant very few things
> go into the branch. I know that pig has pretty decent tests - better
> coverage than many other projects. However, we do not have any testing at
> scale and inevitably, users end up doing testing. So any time we deploy new
> major version, it takes us at least a month to get it stable and once it is
> stabilized we want to keep it this way.
>
> So for us at Yahoo, the only way to work directly from the branch is to go
> by our original plan. If that is not possible, we would go with the private
> git branch.
>
> Olga
>
>
> ________________________________
>  From: Alan Gates <[EMAIL PROTECTED]>
> To: [EMAIL PROTECTED]
> Sent: Friday, November 2, 2012 8:19 PM
> Subject: Re: Our release process
>
> I am all for maintaining stability of branches, and the trunk, as everyone
> benefits from it.  But I do not think this means we should limit bug fixing
> in the branches to only critical issues.  As Pig gets more users we have
> more and more people on older branches who will want fixes for bugs without
> dealing with bigger version changes.  So I am not in favor of limiting
> checkins to branches to P1 issues.
>
> What if we maintain stability on the branches by quickly reverting any
> patches that break the build, the unit tests, or the e2e tests?  This
> allows us to move forward with bug fix versions, it allows those who depend
> on branch stability (which I suspect is everyone in the distribution
> business plus everyone rolling their own Pig), and it should promote
> developer responsibility (no one likes having their patches reverted).
>
> Alan.
>
> On Nov 2, 2012, at 3:58 PM, Olga Natkovich wrote:

Hi,

On Mon, Nov 5, 2012 at 10:48 AM, Olga Natkovich <[EMAIL PROTECTED]> wrote:
> Hi Gianmarco,
>
> Thanks for your comments. Here is a little more information.
>
> At Yahoo, we consider the following issues to be P1:
>
> (1) Bugs that cause wrong results being produced silently
> (2) Bugs that cause failures with no easy workaround
>

Thanks Olga, now I get what you mean.
I don't have a strong opinion on this.
On one hand I see why you don't want to put too many patches in the
branches in order to keep things stable.
On the other hand when we do a 0.10.x release with x>0 the users would
like to have as many bugs fixed as possible.

> Regarding tests. I would suggest we have different rules for trunk and branches:
>
> (1) For branches, I think we should run the full regression suite (including e2e) prior to commit. This way we can ensure branch stability and, as number of patches should be small, will not be a burden
> (2) For trunk, we can go with test-commit only and fix things quickly when things break.

I think this makes sense. +1

> Olga

Cheers,
--
Gianmarco

This seems to make sense to me. People can always back-port features, and
this encourages them to use the newer ones. It also means we will be more
rigorous about stability, which is good as it is a big plus for Pig. I
think for older branches, stability trumps features in a big way.
2012/11/5 Gianmarco De Francisci Morales <[EMAIL PROTECTED]>

> Hi,
>
> On Mon, Nov 5, 2012 at 10:48 AM, Olga Natkovich <[EMAIL PROTECTED]>
> wrote:
> > Hi Gianmarco,
> >
> > Thanks for your comments. Here is a little more information.
> >
> > At Yahoo, we consider the following issues to be P1:
> >
> > (1) Bugs that cause wrong results being produced silently
> > (2) Bugs that cause failures with no easy workaround
> >
>
> Thanks Olga, now I get what you mean.
> I don't have a strong opinion on this.
> On one hand I see why you don't want to put too many patches in the
> branches in order to keep things stable.
> On the other hand when we do a 0.10.x release with x>0 the users would
> like to have as many bugs fixed as possible.
>
> > Regarding tests. I would suggest we have different rules for trunk and
> branches:
> >
> > (1) For branches, I think we should run the full regression suite
> (including e2e) prior to commit. This way we can ensure branch stability
> and, as number of patches should be small, will not be a burden
> > (2) For trunk, we can go with test-commit only and fix things quickly
> when things break.
>
> I think this makes sense. +1
>
> > Olga
>
> Cheers,
> --
> Gianmarco
>

Jonathan, for clarity, are you saying you agree that we should only put bug fixes in branches or we should only put high priority bug fixes in branches?  I think we all agree on the former, but there appear to be different views on the latter.

Alan.

On Nov 5, 2012, at 4:53 PM, Jonathan Coveney wrote:

> This seems to make sense to me. People can always back-port features, and
> this encourages them to use the newer ones. It also means we will be more
> rigorous about stability, which is good as it is a big plus for Pig. I
> think for older branches, stability trumps features in a big way.
>
>
> 2012/11/5 Gianmarco De Francisci Morales <[EMAIL PROTECTED]>
>
>> Hi,
>>
>> On Mon, Nov 5, 2012 at 10:48 AM, Olga Natkovich <[EMAIL PROTECTED]>
>> wrote:
>>> Hi Gianmarco,
>>>
>>> Thanks for your comments. Here is a little more information.
>>>
>>> At Yahoo, we consider the following issues to be P1:
>>>
>>> (1) Bugs that cause wrong results being produced silently
>>> (2) Bugs that cause failures with no easy workaround
>>>
>>
>> Thanks Olga, now I get what you mean.
>> I don't have a strong opinion on this.
>> On one hand I see why you don't want to put too many patches in the
>> branches in order to keep things stable.
>> On the other hand when we do a 0.10.x release with x>0 the users would
>> like to have as many bugs fixed as possible.
>>
>>> Regarding tests. I would suggest we have different rules for trunk and
>> branches:
>>>
>>> (1) For branches, I think we should run the full regression suite
>> (including e2e) prior to commit. This way we can ensure branch stability
>> and, as number of patches should be small, will not be a burden
>>> (2) For trunk, we can go with test-commit only and fix things quickly
>> when things break.
>>
>> I think this makes sense. +1
>>
>>> Olga
>>
>> Cheers,
>> --
>> Gianmarco
>>

I think it might be good to clarify (for me) a couple of cases:

1. we have branched a new release
2. an existing release

The way I understand things, in the case of 1, we have a backlog of patches
(not all of which are P1 bugs), and that's ok. If a new bad bug comes in
(the subject of debate here), then it goes in anyway (and in some cases,
would go into 0.9 etc).

Olga is saying that for existing release (0.9, 0.10), we should only commit
P1 bug fixes there. This makes sense to me, as we're fixing the "official
release" in place.

IMHO, this would encourage people to use newer release (as this is where
the latest and greatest stuff is, including non-critical bug fixes). Olga's
criteria is a pretty clear barrier for inclusion into these releases. With
old releases, I think the key is really that they keep doing what they have
always done. Most bugs are well understood by now, and the ones that aren't
will no doubt be P1.

I'm not decided (thus no formal +1 or whatnot), but Olga's point seems
pretty reasonable to me, especially given that trunk has pretty liberal
development. Once it gets tidied up, I can understand not wanting to jostle
it.
2012/11/5 Alan Gates <[EMAIL PROTECTED]>

> Jonathan, for clarity, are you saying you agree that we should only put
> bug fixes in branches or we should only put high priority bug fixes in
> branches?  I think we all agree on the former, but there appear to be
> different views on the latter.
>
> Alan.
>
> On Nov 5, 2012, at 4:53 PM, Jonathan Coveney wrote:
>
> > This seems to make sense to me. People can always back-port features, and
> > this encourages them to use the newer ones. It also means we will be more
> > rigorous about stability, which is good as it is a big plus for Pig. I
> > think for older branches, stability trumps features in a big way.
> >
> >
> > 2012/11/5 Gianmarco De Francisci Morales <[EMAIL PROTECTED]>
> >
> >> Hi,
> >>
> >> On Mon, Nov 5, 2012 at 10:48 AM, Olga Natkovich <[EMAIL PROTECTED]>
> >> wrote:
> >>> Hi Gianmarco,
> >>>
> >>> Thanks for your comments. Here is a little more information.
> >>>
> >>> At Yahoo, we consider the following issues to be P1:
> >>>
> >>> (1) Bugs that cause wrong results being produced silently
> >>> (2) Bugs that cause failures with no easy workaround
> >>>
> >>
> >> Thanks Olga, now I get what you mean.
> >> I don't have a strong opinion on this.
> >> On one hand I see why you don't want to put too many patches in the
> >> branches in order to keep things stable.
> >> On the other hand when we do a 0.10.x release with x>0 the users would
> >> like to have as many bugs fixed as possible.
> >>
> >>> Regarding tests. I would suggest we have different rules for trunk and
> >> branches:
> >>>
> >>> (1) For branches, I think we should run the full regression suite
> >> (including e2e) prior to commit. This way we can ensure branch stability
> >> and, as number of patches should be small, will not be a burden
> >>> (2) For trunk, we can go with test-commit only and fix things quickly
> >> when things break.
> >>
> >> I think this makes sense. +1
> >>
> >>> Olga
> >>
> >> Cheers,
> >> --
> >> Gianmarco
> >>
>
>

The push for an upgrade will work only if the higher release is backward compatible with the lower release. If not, folks will tend to use private branches. Having a stable branch on a large deployment is a good indicator of stability. However, please note that there have been instances where some releases were never adopted. I will be extremely careful in applying the rule of running e2e tests for every commit to a released branch.

If we release every quarter (hopefully) and preserve backward compatibility then I am +1 to the proposal. If the backward compatibility is not preserved then I am -1 for having to run e2e for every commit to a released branch.

Santhosh
________________________________
 From: Jonathan Coveney <[EMAIL PROTECTED]>
To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>
Sent: Tuesday, November 6, 2012 6:34 PM
Subject: Re: Our release process
 
I think it might be good to clarify (for me) a couple of cases:

1. we have branched a new release
2. an existing release

The way I understand things, in the case of 1, we have a backlog of patches
(not all of which are P1 bugs), and that's ok. If a new bad bug comes in
(the subject of debate here), then it goes in anyway (and in some cases,
would go into 0.9 etc).

Olga is saying that for existing release (0.9, 0.10), we should only commit
P1 bug fixes there. This makes sense to me, as we're fixing the "official
release" in place.

IMHO, this would encourage people to use newer release (as this is where
the latest and greatest stuff is, including non-critical bug fixes). Olga's
criteria is a pretty clear barrier for inclusion into these releases. With
old releases, I think the key is really that they keep doing what they have
always done. Most bugs are well understood by now, and the ones that aren't
will no doubt be P1.

I'm not decided (thus no formal +1 or whatnot), but Olga's point seems
pretty reasonable to me, especially given that trunk has pretty liberal
development. Once it gets tidied up, I can understand not wanting to jostle
it.
2012/11/5 Alan Gates <[EMAIL PROTECTED]>

> Jonathan, for clarity, are you saying you agree that we should only put
> bug fixes in branches or we should only put high priority bug fixes in
> branches?  I think we all agree on the former, but there appear to be
> different views on the latter.
>
> Alan.
>
> On Nov 5, 2012, at 4:53 PM, Jonathan Coveney wrote:
>
> > This seems to make sense to me. People can always back-port features, and
> > this encourages them to use the newer ones. It also means we will be more
> > rigorous about stability, which is good as it is a big plus for Pig. I
> > think for older branches, stability trumps features in a big way.
> >
> >
> > 2012/11/5 Gianmarco De Francisci Morales <[EMAIL PROTECTED]>
> >
> >> Hi,
> >>
> >> On Mon, Nov 5, 2012 at 10:48 AM, Olga Natkovich <[EMAIL PROTECTED]>
> >> wrote:
> >>> Hi Gianmarco,
> >>>
> >>> Thanks for your comments. Here is a little more information.
> >>>
> >>> At Yahoo, we consider the following issues to be P1:
> >>>
> >>> (1) Bugs that cause wrong results being produced silently
> >>> (2) Bugs that cause failures with no easy workaround
> >>>
> >>
> >> Thanks Olga, now I get what you mean.
> >> I don't have a strong opinion on this.
> >> On one hand I see why you don't want to put too many patches in the
> >> branches in order to keep things stable.
> >> On the other hand when we do a 0.10.x release with x>0 the users would
> >> like to have as many bugs fixed as possible.
> >>
> >>> Regarding tests. I would suggest we have different rules for trunk and
> >> branches:
> >>>
> >>> (1) For branches, I think we should run the full regression suite
> >> (including e2e) prior to commit. This way we can ensure branch stability
> >> and, as number of patches should be small, will not be a burden
> >>> (2) For trunk, we can go with test-commit only and fix things quickly
> >> when things break.
> >>
> >> I think this makes sense. +1
> >>
> >>> Olga
> >>
> >> Cheers,

Hi Santhosh,

Can you clarify why running e2e tests on every checking is a problem?

Olga
________________________________
From: Santhosh M S <[EMAIL PROTECTED]>
To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>
Sent: Monday, November 19, 2012 3:48 PM
Subject: Re: Our release process

The push for an upgrade will work only if the higher release is backward compatible with the lower release. If not, folks will tend to use private branches. Having a stable branch on a large deployment is a good indicator of stability. However, please note that there have been instances where some releases were never adopted. I will be extremely careful in applying the rule of running e2e tests for every commit to a released branch.

If we release every quarter (hopefully) and preserve backward compatibility then I am +1 to the proposal. If the backward compatibility is not preserved then I am -1 for having to run e2e for every commit to a released branch.

Santhosh
________________________________
From: Jonathan Coveney <[EMAIL PROTECTED]>
To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>
Sent: Tuesday, November 6, 2012 6:34 PM
Subject: Re: Our release process

I think it might be good to clarify (for me) a couple of cases:

1. we have branched a new release
2. an existing release

The way I understand things, in the case of 1, we have a backlog of patches
(not all of which are P1 bugs), and that's ok. If a new bad bug comes in
(the subject of debate here), then it goes in anyway (and in some cases,
would go into 0.9 etc).

Olga is saying that for existing release (0.9, 0.10), we should only commit
P1 bug fixes there. This makes sense to me, as we're fixing the "official
release" in place.

IMHO, this would encourage people to use newer release (as this is where
the latest and greatest stuff is, including non-critical bug fixes). Olga's
criteria is a pretty clear barrier for inclusion into these releases. With
old releases, I think the key is really that they keep doing what they have
always done. Most bugs are well understood by now, and the ones that aren't
will no doubt be P1.

I'm not decided (thus no formal +1 or whatnot), but Olga's point seems
pretty reasonable to me, especially given that trunk has pretty liberal
development. Once it gets tidied up, I can understand not wanting to jostle
it.
2012/11/5 Alan Gates <[EMAIL PROTECTED]>

> Jonathan, for clarity, are you saying you agree that we should only put
> bug fixes in branches or we should only put high priority bug fixes in
> branches?  I think we all agree on the former, but there appear to be
> different views on the latter.
>
> Alan.
>
> On Nov 5, 2012, at 4:53 PM, Jonathan Coveney wrote:
>
> > This seems to make sense to me. People can always back-port features, and
> > this encourages them to use the newer ones. It also means we will be more
> > rigorous about stability, which is good as it is a big plus for Pig. I
> > think for older branches, stability trumps features in a big way.
> >
> >
> > 2012/11/5 Gianmarco De Francisci Morales <[EMAIL PROTECTED]>
> >
> >> Hi,
> >>
> >> On Mon, Nov 5, 2012 at 10:48 AM, Olga Natkovich <[EMAIL PROTECTED]>
> >> wrote:
> >>> Hi Gianmarco,
> >>>
> >>> Thanks for your comments. Here is a little more information.
> >>>
> >>> At Yahoo, we consider the following issues to be P1:
> >>>
> >>> (1) Bugs that cause wrong results being produced silently
> >>> (2) Bugs that cause failures with no easy workaround
> >>>
> >>
> >> Thanks Olga, now I get what you mean.
> >> I don't have a strong opinion on this.
> >> On one hand I see why you don't want to put too many patches in the
> >> branches in order to keep things stable.
> >> On the other hand when we do a 0.10.x release with x>0 the users would
> >> like to have as many bugs fixed as possible.
> >>
> >>> Regarding tests. I would suggest we have different rules for trunk and
> >> branches:
> >>>
> >>> (1) For branches, I think we should run the full regression suite
> >> (including e2e) prior to commit. This way we can ensure branch stability

It takes too long to run. If the e2e tests are run every night or a reasonable timeframe then it will reduce the barrier for submitting patches. The context for this: the reluctance of folks to move to a higher version when the higher version is not backward compatible.

Santhosh
________________________________
 From: Olga Natkovich <[EMAIL PROTECTED]>
To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>; Santhosh M S <[EMAIL PROTECTED]>
Sent: Sunday, November 25, 2012 5:56 PM
Subject: Re: Our release process
 
Hi Santhosh,

Can you clarify why running e2e tests on every checking is a problem?

Olga
________________________________
From: Santhosh M S <[EMAIL PROTECTED]>
To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>
Sent: Monday, November 19, 2012 3:48 PM
Subject: Re: Our release process

The push for an upgrade will work only if the higher release is backward compatible with the lower release. If not, folks will tend to use private branches. Having a stable branch on a large deployment is a good indicator of stability. However, please note that there have been instances where some releases were never adopted. I will be extremely careful in applying the rule of running e2e tests for every commit to a released branch.

If we release every quarter (hopefully) and preserve backward compatibility then I am +1 to the proposal. If the backward compatibility is not preserved then I am -1 for having to run e2e for every commit to a released branch.

Santhosh
________________________________
From: Jonathan Coveney <[EMAIL PROTECTED]>
To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>
Sent: Tuesday, November 6, 2012 6:34 PM
Subject: Re: Our release process

I think it might be good to clarify (for me) a couple of cases:

1. we have branched a new release
2. an existing release

The way I understand things, in the case of 1, we have a backlog of patches
(not all of which are P1 bugs), and that's ok. If a new bad bug comes in
(the subject of debate here), then it goes in anyway (and in some cases,
would go into 0.9 etc).

Olga is saying that for existing release (0.9, 0.10), we should only commit
P1 bug fixes there. This makes sense to me, as we're fixing the "official
release" in place.

IMHO, this would encourage people to use newer release (as this is where
the latest and greatest stuff is, including non-critical bug fixes). Olga's
criteria is a pretty clear barrier for inclusion into these releases. With
old releases, I think the key is really that they keep doing what they have
always done. Most bugs are well understood by now, and the ones that aren't
will no doubt be P1.

I'm not decided (thus no formal +1 or whatnot), but Olga's point seems
pretty reasonable to me, especially given that trunk has pretty liberal
development. Once it gets tidied up, I can understand not wanting to jostle
it.
2012/11/5 Alan Gates <[EMAIL PROTECTED]>

> Jonathan, for clarity, are you saying you agree that we should only put
> bug fixes in branches or we should only put high priority bug fixes in
> branches?  I think we all agree on the former, but there appear to be
> different views on the latter.
>
> Alan.
>
> On Nov 5, 2012, at 4:53 PM, Jonathan Coveney wrote:
>
> > This seems to make sense to me. People can always back-port features, and
> > this encourages them to use the newer ones. It also means we will be more
> > rigorous about stability, which is good as it is a big plus for Pig. I
> > think for older branches, stability trumps features in a big way.
> >
> >
> > 2012/11/5 Gianmarco De Francisci Morales <[EMAIL PROTECTED]>
> >
> >> Hi,
> >>
> >> On Mon, Nov 5, 2012 at 10:48 AM, Olga Natkovich <[EMAIL PROTECTED]>
> >> wrote:
> >>> Hi Gianmarco,
> >>>
> >>> Thanks for your comments. Here is a little more information.
> >>>
> >>> At Yahoo, we consider the following issues to be P1:
> >>>
> >>> (1) Bugs that cause wrong results being produced silently
> >>> (2) Bugs that cause failures with no easy workaround
> >>>
> >>
>

Hi Santhosh,

I understand the compatibility issue though I am not sure we can guarantee it for all releases upfront but agree that we should make an effort.

On the e2e tests, part of the proposal is only do make P1 type of changes to the branch after the initial release so they should be rare.

Olga
________________________________
 From: Santhosh M S <[EMAIL PROTECTED]>
To: Olga Natkovich <[EMAIL PROTECTED]>; "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>
Sent: Monday, November 26, 2012 12:00 AM
Subject: Re: Our release process
 

It takes too long to run. If the e2e tests are run every night or a reasonable timeframe then it will reduce the barrier for submitting patches. The context for this: the reluctance of folks to move to a higher version when the higher version is not backward compatible.

Santhosh
________________________________
 From: Olga Natkovich <[EMAIL PROTECTED]>
To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>; Santhosh M S <[EMAIL PROTECTED]>
Sent: Sunday, November 25, 2012 5:56 PM
Subject: Re: Our release process
 
Hi Santhosh,

Can you clarify why running e2e tests on every checking is a problem?

Olga
________________________________
From: Santhosh M S <[EMAIL PROTECTED]>
To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>
Sent: Monday, November 19, 2012 3:48 PM
Subject: Re: Our release process

The push for an upgrade will work only if the higher release is backward compatible with the lower release. If not, folks will tend to use private branches. Having a stable branch on a large deployment is a good indicator of stability. However, please note that there have been instances where some releases were never adopted. I will be extremely careful in applying the rule of
 running e2e tests for every commit to a released branch.

If we release every quarter (hopefully) and preserve backward compatibility then I am +1 to the proposal. If the backward compatibility is not preserved then I am -1 for having to run e2e for every commit to a released branch.

Santhosh
________________________________
From: Jonathan Coveney <[EMAIL PROTECTED]>
To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>
Sent: Tuesday, November 6, 2012 6:34 PM
Subject: Re: Our release process

I think it might be good to clarify (for me) a couple of cases:

1. we have branched a new release
2. an existing release

The way I understand things, in the case of 1, we have
 a backlog of patches
(not all of which are P1 bugs), and that's ok. If a new bad bug comes in
(the subject of debate here), then it goes in anyway (and in some cases,
would go into 0.9 etc).

Olga is saying that for existing release (0.9, 0.10), we should only commit
P1 bug fixes there. This makes sense to me, as we're fixing the "official
release" in place.

IMHO, this would encourage people to use newer release (as this is where
the latest and greatest stuff is, including non-critical bug fixes). Olga's
criteria is a pretty clear barrier for inclusion into these releases. With
old releases, I think the key is really that they keep doing what they have
always done. Most bugs are well understood by now, and the ones that aren't
will no doubt be P1.

I'm not decided (thus no formal +1 or whatnot), but Olga's point seems
pretty reasonable to me, especially given that trunk has pretty
 liberal
development. Once it gets tidied up, I can understand not wanting to jostle
it.
2012/11/5 Alan Gates <[EMAIL PROTECTED]>

> Jonathan, for clarity, are you saying you agree that we should only put
> bug fixes in branches or we should only put high priority bug fixes in
> branches?  I think we all agree on the former, but there appear to be
> different views on the latter.
>
> Alan.
>
> On Nov 5, 2012, at 4:53 PM, Jonathan Coveney wrote:
>
> > This seems to make sense to me. People can always back-port features, and
> > this encourages them to use the newer ones. It also means we will be more
> > rigorous about stability, which is good as it is a big plus for Pig. I
 >
 this.
 >>

Looks like everyone is interested in having frequent releases - I don't see anyone disagreeing with that.

Regarding "If a patch makes the release branch unstable, we revert it" - what are the criteria? If we can't decide on the criteria on this thread (already pretty long) then lets get the release trains going. We can revisit the criteria for inclusion of bug fixes when that happens.

Santhosh
________________________________
 From: Julien Le Dem <[EMAIL PROTECTED]>
To: [EMAIL PROTECTED]; Santhosh M S <[EMAIL PROTECTED]>
Cc: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>
Sent: Thursday, November 29, 2012 9:45 AM
Subject: Re: Our release process
 
The release branch receives only bug fixes. Patch level releases (3rd
version number) are issued out of the release branch and introduce
only bug fixes and no new features.
Deciding whether a patch is applied to the release branch is based on
preserving stability (as Bill said). If a patch makes the release
branch unstable, we revert it.
New features are added to trunk where new major and minor releases will happen.
If we need a new feature out then we make a new minor release.
Doing frequent releases is the industry standard and will resolve
conflicts around what should go in a release branch.

Making a new release is currently painful *because* we wait so long in
between two releases. Let's fix that.

Julien

On Wed, Nov 28, 2012 at 10:09 PM, Santhosh M S
<[EMAIL PROTECTED]> wrote:
> Since releasing a major version once a month is agressive and we have not released on a quarterly basis, we should allow commits to a released branch to facilitate dot releases.
>
> If we are allowing commits to a released branch, the criteria for inclusion can be created anew or we use the industry standards for severity (or priority). It could be painful for a few folks but I don't see better alternatives.
>
> Regarding reverting commits based on e2e tests breaking:
>         1. Who is running the tests?
>         2. How often are they run?
> If we have nightly e2e runs then its easier to catch these errors early. If not the barrier for inclusion is pretty high and time consuming making it harder to develop.
>
> Santhosh
>
>
> ________________________________
>  From: Bill Graham <[EMAIL PROTECTED]>
> To: [EMAIL PROTECTED]
> Sent: Wednesday, November 28, 2012 11:39 AM
> Subject: Re: Our release process
>
> I agree releasing often is ideal, but releasing major versions once a month
> would be a bit agressive.
>
> +1 to Olga's initial definition of how Yahoo! determines what goes into a
> released branch. Basically is something broken without a workaround or is
> there potential silent data loss. Trying to get a more granular definition
> than that (i.e. P1, P2, severity, etc) will be painful. The reality in that
> case is that for whomever is blocked by the bug will consider it a P1.
>
> Fixes need to be relatively low-risk though to keep stability, but this is
> also subjective. For this I'm in favor of relying on developer and reviewer
> judgement to make that call and I'm +1 to Alan's proposal of rolling back
> patches that break the e2e tests or anything else.
>
> I think our policy should avoid time-based consideration on how many
> quarters away are we from the next major release since that's also
> impossible to quantify. Plus, if the answer to the question is that we're
> more than 1-2 quarters from the next release is "yes" then we should be
> fixing that release problem.
>
>
> On Wed, Nov 28, 2012 at 10:22 AM, Julien Le Dem <[EMAIL PROTECTED]> wrote:
>
>> I would really like to see us doing frequent releases (at least once
>> per quarter if not once a month).
>> I think the whole notion of priority or being a "blocker" is subjective.
>> Releasing infrequently pressures us to push more changes than we would
>> want to the release branch.
>> We should focus on keeping TRUNK stable as well so that it is easier
>> to release and users can do more frequent and smaller upgrades.

Proposed criteria:
 - it makes the tests fail. targets test-commit + test + e2e tests
 - a critical bug is reported in a short time frame (definition of
critical not needed as it is rare and can be decided on a case by case
basis)

That raises another question: what are the existing CI servers running
the tests?
 - the Apache CI runs test-commit and test (is it more stable now?)
and not e2e. It would be great if it did.
 - we have a Jenkins build at Twitter where we run test-commit and
test, we could not run e2e easily in our environment.
 - I understand there's a Yahoo/Hortonworks build (test-commit + test + e2e ???)

Whenever those builds fail we should open or reopen JIRAS and fix it.

The time it takes to run the full test suite makes it impractical to
run on a desktop/laptop.

For the release Pig-0.11.0 we need to get this list of JIRAs down to 0
and publish the jar.
https://issues.apache.org/jira/secure/IssueNavigator.jspa?reset=true&jqlQuery=project+%3D+PIG+AND+fixVersion+%3D+%220.11%22+AND+resolution+%3D+Unresolved+ORDER+BY+updated+DESC%2C+due+ASC%2C+priority+DESC

Julien

On Thu, Nov 29, 2012 at 11:16 PM, Santhosh M S
<[EMAIL PROTECTED]> wrote:
> Looks like everyone is interested in having frequent releases - I don't see anyone disagreeing with that.
>
> Regarding "If a patch makes the release branch unstable, we revert it" - what are the criteria? If we can't decide on the criteria on this thread (already pretty long) then lets get the release trains going. We can revisit the criteria for inclusion of bug fixes when that happens.
>
> Santhosh
>
>
> ________________________________
>  From: Julien Le Dem <[EMAIL PROTECTED]>
> To: [EMAIL PROTECTED]; Santhosh M S <[EMAIL PROTECTED]>
> Cc: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>
> Sent: Thursday, November 29, 2012 9:45 AM
> Subject: Re: Our release process
>
> The release branch receives only bug fixes. Patch level releases (3rd
> version number) are issued out of the release branch and introduce
> only bug fixes and no new features.
> Deciding whether a patch is applied to the release branch is based on
> preserving stability (as Bill said). If a patch makes the release
> branch unstable, we revert it.
> New features are added to trunk where new major and minor releases will happen.
> If we need a new feature out then we make a new minor release.
> Doing frequent releases is the industry standard and will resolve
> conflicts around what should go in a release branch.
>
> Making a new release is currently painful *because* we wait so long in
> between two releases. Let's fix that.
>
> Julien
>
> On Wed, Nov 28, 2012 at 10:09 PM, Santhosh M S
> <[EMAIL PROTECTED]> wrote:
>> Since releasing a major version once a month is agressive and we have not released on a quarterly basis, we should allow commits to a released branch to facilitate dot releases.
>>
>> If we are allowing commits to a released branch, the criteria for inclusion can be created anew or we use the industry standards for severity (or priority). It could be painful for a few folks but I don't see better alternatives.
>>
>> Regarding reverting commits based on e2e tests breaking:
>>         1. Who is running the tests?
>>         2. How often are they run?
>> If we have nightly e2e runs then its easier to catch these errors early. If not the barrier for inclusion is pretty high and time consuming making it harder to develop.
>>
>> Santhosh
>>
>>
>> ________________________________
>>  From: Bill Graham <[EMAIL PROTECTED]>
>> To: [EMAIL PROTECTED]
>> Sent: Wednesday, November 28, 2012 11:39 AM
>> Subject: Re: Our release process
>>
>> I agree releasing often is ideal, but releasing major versions once a month
>> would be a bit agressive.
>>
>> +1 to Olga's initial definition of how Yahoo! determines what goes into a
>> released branch. Basically is something broken without a workaround or is
>> there potential silent data loss. Trying to get a more granular definition
>> than that (i.e. P1, P2, severity, etc) will be painful. The reality in that

HI Julien,

You are making most of the points that I did on this thread (CI for e2e, not burdening clean e2e prior to every commit for a release branch). The only point on which there is no clear agreement is the definition of a bug that can be included in a previously released branch. I am fine with a case by case inclusion. 

Hi Olga,

Are you fine with Julien's proposal as it stands - bugs that are included will be determined at the time of inclusion instead of doing it now.

Santhosh
________________________________
 From: Julien Le Dem <[EMAIL PROTECTED]>
To: [EMAIL PROTECTED]; Santhosh M S <[EMAIL PROTECTED]>
Cc: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>
Sent: Friday, November 30, 2012 5:37 PM
Subject: Re: Our release process
 
Proposed criteria:
- it makes the tests fail. targets test-commit + test + e2e tests
- a critical bug is reported in a short time frame (definition of
critical not needed as it is rare and can be decided on a case by case
basis)

That raises another question: what are the existing CI servers running
the tests?
- the Apache CI runs test-commit and test (is it more stable now?)
and not e2e. It would be great if it did.
- we have a Jenkins build at Twitter where we run test-commit and
test, we could not run e2e easily in our environment.
- I understand there's a Yahoo/Hortonworks build (test-commit + test + e2e ???)

Whenever those builds fail we should open or reopen JIRAS and fix it.

The time it takes to run the full
 test suite makes it impractical to
run on a desktop/laptop.

For the release Pig-0.11.0 we need to get this list of JIRAs down to 0
and publish the jar.
https://issues.apache.org/jira/secure/IssueNavigator.jspa?reset=true&jqlQuery=project+%3D+PIG+AND+fixVersion+%3D+%220.11%22+AND+resolution+%3D+Unresolved+ORDER+BY+updated+DESC%2C+due+ASC%2C+priority+DESC

Julien

On Thu, Nov 29, 2012 at 11:16 PM, Santhosh M S
<[EMAIL PROTECTED]> wrote:
> Looks like everyone is interested in having frequent releases - I don't see anyone disagreeing with that.
>
> Regarding "If a patch
 makes the release branch unstable, we revert it" - what are the criteria? If we can't decide on the criteria on this thread (already pretty long) then lets get the release trains going. We can revisit the criteria for inclusion of bug fixes when that happens.
>
> Santhosh
>
>
> ________________________________
>  From: Julien Le Dem <[EMAIL PROTECTED]>
> To: [EMAIL PROTECTED]; Santhosh M S <[EMAIL PROTECTED]>
> Cc: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>
> Sent:
 Thursday, November 29, 2012 9:45 AM
> Subject: Re: Our release process
>
> The release branch receives only bug fixes. Patch level releases (3rd
> version number) are issued out of the release branch and introduce
> only bug fixes and no new features.
> Deciding whether a patch is applied to the release branch is based on
> preserving stability (as Bill said). If a patch makes the release
> branch unstable, we revert it.
> New features are added to trunk where new major and minor releases will happen.
> If we need a new feature out then we make a new minor release.
> Doing frequent releases is the industry standard and will resolve
> conflicts around what should go in a release branch.
>
> Making a new release is currently painful *because* we wait so long in
> between two releases. Let's fix that.
>
> Julien
>
> On Wed, Nov 28, 2012 at
 10:09 PM, Santhosh M S
> <[EMAIL PROTECTED]> wrote:
>> Since releasing a major version once a month is agressive and we have not released on a quarterly basis, we should allow commits to a released branch to facilitate dot releases.
>>
>> If we are allowing commits to a released branch, the criteria for inclusion can be created anew or we use the industry standards for severity (or priority). It could be painful for a few folks but I don't see better alternatives.
>>
>> Regarding reverting commits based on e2e tests breaking:
>>         1. Who is running the tests?
>>         2. How often are they run?
 consuming making it harder to develop.
 painful. The reality in that
 often.
 skipping the other levels (normal, minor and trivial) for this
 >
 Message -----
 >
 >
 the reluctance of folks to move to a higher
 Santhosh,
 private
 cases,
 > I'm not decided (thus no formal +1 or whatnot), but Olga's point seems
 >>
 Natkovich <
 this.
 >> >>> (2) For trunk, we can go with test-commit only and fix things