As we're starting to consider 1.5.2 and 1.6.1 coming out in the near future, I want to revisit a discussion I started at the end of April regarding the "testing burden" that is currently set forth in our release document.
What I'm proposing is to modify the language of the release document to be explicit about the amount of testing needed. For bug-fix, "minor" releases (e.g. 1.5.2 and 1.6.1), the 7 days of testing using continuous ingest and randomwalk (with and without agitation) will be clearly defined as "may" instead of "should" or "must" language. If the resources are available, it is recommended that some longer, multi-process/node test is run against the release candidate; however, it is not required and should not prevent us from making the minor release.
I will also include language that strongly recommends that the changes included in the "minor" release be vetted/reviewed as a way to mitigate the risk of shipping new regressions.
I am not recommending that the language be changed for "major" releases (e.g. 1.7.0 and 2.0.0) as these releases still imply significant new features or internal changes.
Unless someone informs me otherwise, I will treat this as a normal lazy-consensus approval. Assuming we move closer to "proper" semantic versioning for 2.0.0, I believe these updated guidelines will change again. I do however think there is merit in making this change now so that we can get the good bugs that we've fixed out to our users.
Let me know what you think. I will wait, at least, the prescribed three days before changing any thing.
+1 for reduced testing burden for bugfixes and a change in the guidelines to reflect that. Christopher L Tubbs II http://gravatar.com/ctubbsii On Thu, Jun 19, 2014 at 3:13 PM, Josh Elser <[EMAIL PROTECTED]> wrote:
-1 I hesitate to step into this discussion because I can't also step up and do the long-term testing even as I recommend that it must be done. There are at least four companies supporting Accumulo and contributing back to the project. Surely one of those companies can supply the resources to continue the existing test regimen? Is there some concern that those resources won't be available for the next release cycle?
On Thu, Jun 19, 2014 at 3:13 PM, Josh Elser <[EMAIL PROTECTED]> wrote:
On Thu, Jun 19, 2014 at 3:13 PM, Josh Elser <[EMAIL PROTECTED]> wrote: I am in favor of lowering the required threshhold. If someone feels not enough testing was done, they can vote -1 the release vote.
I don't know Josh's concerns, but the concern for me is both resources and time. No matter how much resources we have, it is still not infinite, and I'd rather we focus our testing efforts on the changeset between the previous release and the minor/bugfix release, rather than spend resources and time on all the exhaustive general testing, which mostly exercises code that has not changed.
For instance, do we really need 72-hours of continuous ingest on a large cluster to release a bugfix which affects the shell? If the long running tests are what is necessary to exercise the changeset, that makes sense, but otherwise, no.
I realize we're past window here, but a few things: 1) I'm concerned about changing our testing standards prior to getting some practice in on holding to tighter bounds on what can go into a particular kind of fix number. However, I generally like the idea of just relying on people voting -1 for insufficient testing. 2) We really could have used a [VOTE] on changing 1.x versions to use bugfix for the last number. It would be an opportunity to formally adopt our release governance docs into the PMC bylaws by reference. 3) As a point of principle, David's veto was completely valid as it stood. While we are individuals it's perfectly reasonable for the PMC to set requirements on what goes into a release we sign off on, even if the PMC members may not always have ready access to the resources needed to meet that standard.
There's no reason David should have to volunteer resources to back up his veto, especially when it was merely calling for a continuation of the standard we already had set.
4) While I'm not in a position to obligate Cloudera, the team I'm on currently has been allocated sufficient resources to meet the current testing standards for a release and I have no reason to believe we won't have said resources when future release windows happen.
On Thu, Jun 19, 2014 at 3:03 PM, David Medinets <[EMAIL PROTECTED]> wrote: Sean
It's ok. I only remembered about this early today. I don't see this as a huge issue since everyone seemed to be in agreement on it. Writing it down for reference would be good for future discussions on the subject. That's why I said "Unless you are willing to supply the funds to pay to use some resources, I don't feel like this is a valid -1." If he, or anyone, is willing provide general resources for testing, that's a different story. Given his response, I assume that is not the case. This would require time from you, Mike or Bill H, yes? While having some dedicated resources is nice, I'm worried about relying on other people (who have their own obligations) to fulfill the release manager's obligations.
I think that gets into the details which the release manager can coordinate and other devs can express their concerns via the normal release voting process.
inline On Mon, Jun 23, 2014 at 2:52 PM, Josh Elser <[EMAIL PROTECTED]> wrote: I don't think everyone is in agreement on it. To wit, I remain opposed to it.
But that's the opposite of what I just said. The opposition and the ability to find funds are not strongly coupled.
The lack of funds would hopefully be a convincing argument to try to sway someone that we should lower the testing barrier, but they aren't a legitimate excuse for invalidating their vote. What if Hypothetical David wanted to do fund raising to get resources together? Would you decide on a deadline that would allow his vote to be legitimate or not?
The basis for a veto need merely be technical. "We've done this level of testing before and it protects our users. We should continue doing it." is a perfectly reasonable justification. (I admit I am taking some liberty in making parts of David's previous concern more explicit)
BTW, situations like this are a part of why Majority Vote for governance decisions are my preference. While I fully agree with David's ability to veto I also agree that the community should be able to override him if no funding source could be found.
The release manager's only responsibility is to ensure the testing gets done, not to do the testing themselves. That's why we're a community: so we can rely on each other.
If a particular release manager needs help making sure the coverage happens, they should just ask for that help when making the release plan. Mike, Bill, and I are all capable of getting approval to block out time in support of things that are good for Apache Accumulo.
(I am not actually sure it would require time from Mike, Bill, or me; circumstances happen and "it depends." It's certainly easier for Mike, Bill, or me to do it.)
Oh it is. I apologize, I misread your initial statement. I wouldn't have a problem with anyone trying to raise funds for test resources, within reason. But, unless someone is actually planning to do this, it probably isn't much more than a hypothetical argument that we need to spend time discussing.
I've update the language on the release guidelines on the staged site for what was discussed here.
Please let me know if there are any problems with it, otherwise I'll push the changes to production later today. If you miss this message before I push to production, please feel free to continue the discussion and we can revise the language and make sure we're all happy.
as a follow up, can we agree on a label or add a flag to jiras that can be used to signal when we think a particular issue introduces changes that warrant the higher level of testing scrutiny? On Fri, Jun 27, 2014 at 12:56 PM, Josh Elser <[EMAIL PROTECTED]> wrote:
There's a label "needs-test" that we don't already use for anything. We could use this to mean "needs more than bugfix level testing."
I started looking at what it takes to add a field, but it's not obvious at first glance. If we figure it out, we could add a "test level" field that says "bugfix", "minor", "major" to indicate how much testing we need. Even if we currently don't define all three levels, where ever we explain the use of the field we could conflate as we desire.
On Fri, Jun 27, 2014 at 2:34 PM, Josh Elser <[EMAIL PROTECTED]> wrote:
I like labels to identify testing needs, as well as which tests were performed. "needs-test", followed by "tested-24hr-ci", etc. Christopher L Tubbs II http://gravatar.com/ctubbsii On Fri, Jun 27, 2014 at 3:44 PM, Sean Busbey <[EMAIL PROTECTED]> wrote:
On Fri, Jun 27, 2014 at 4:34 PM, Christopher <[EMAIL PROTECTED]> wrote: oh, yeeeeeessss. that would be nice. that would give us a better idea of testing coverage without requiring explicit coordination pre-RC.
NEW: Monitor These Apps!
Apache Lucene, Apache Solr and all other Apache Software Foundation projects and their respective logos are trademarks of the Apache Software Foundation.
Elasticsearch, Kibana, Logstash, and Beats are trademarks of Elasticsearch BV, registered in the U.S. and in other countries. This site and Sematext Group is in no way affiliated with Elasticsearch BV.
Service operated by Sematext