Anecdotally, I think I have observed the following pattern of activity on the Avro project:
1. Not much happens for several months.
2. Doug announces that he wants to cut a release candidate soon.
3. Frenzied activity arises as everybody tries to get their patches into the release.
4. The release happens much later than planned, due to there being so many things that want to get in.
For example, 1.7.7 was planned for "next week" over a month ago; 1.7.6 was planned for "later this week" but took almost 6 weeks. The time between releases was about 6 months.
Whilst it's great that the prospect of a release can stimulate so much activity, I think it would be good for the health of the Avro project, and good for our users, if releases were smaller and more frequent. By all means we should keep the same backwards-compatibility rules; but if there's a simple bugfix, we should be able to get it out sooner.
In particular, the currently-released version of Avro for Ruby (1.7.6) is totally broken -- it won't even install. (Major problems are https://issues.apache.org/jira/browse/AVRO-1499 and https://issues.apache.org/jira/browse/AVRO-1459 -- both have been fixed and are just awaiting release). I'm embarrassed to tell my Ruby friends about Avro, because I have to preface it with "I'm afraid it doesn't actually work". Consequently, there has been a fork (https://github.com/wvanbergen/tros) with the express purpose of getting away from the Apache Avro release schedule.
It doesn't have to be this way. Please may I suggest the following:
(1) We aim to do point releases regularly, e.g. 1 or 2 releases per month, even if there haven't been many commits in the meantime. As these will be small incremental releases, there will be no rush to get a patch into this release, because you know that the next release isn't far away. This ensures that bugfixes aren't withheld from users unnecessarily.
(2) If voting on releases so frequently is too much of a burden, we could decouple the point version numbers for the different language implementations, so that different languages can release independently, and only require voting on those that have changed. (I'm not sure whether this is a good idea, or even compatible with ASF policy.)
What do you think?
-----BEGIN PGP SIGNATURE-----
On Thu, Jun 26, 2014 at 5:02 AM, Martin Kleppmann <[EMAIL PROTECTED]> wrote:
More frequent Avro releases would be great. I can't personally commit
to increasing release cadence, as my availability is intermittent.
But any committer can create releases. Configuring a machine so that
all languages are built and tested takes a bit of time but isn't too
difficult. I'd love to see someone else take on this task.
This is technically feasible. Currently the first part of an Avro
version indicates the Avro schema & encoding version, the second part
indicates the API/runtime version, and the third the bugfix version.
So in theory each language can increment the second and third parts
independently. We might need to better advertise that, e.g.,
avro-java-1.7.7 works fine with avro-ruby-1.9.3, that only a match on
the first digit is required for data compatibility.
If we decide to go this way we need to reorganize sources into a trunk
& branches per language, update build files accordingly, then have
folks step up and volunteer to release each language. It would
probably increase the overall effort spent releasing. (Once you have
all the dependencies installed, creating a release for all languages
isn't hard.) So, again, whether this works depends on the level of
volunteer effort available. If enough folks volunteer for this
approach then I'd be happy to see it implemented.
On Thu, Jun 26, 2014 at 10:29 AM, Doug Cutting <[EMAIL PROTECTED]> wrote:
How much of this work can a non-committer take on?
On Thu, Jun 26, 2014 at 8:59 AM, Sean Busbey <[EMAIL PROTECTED]> wrote:
If you take on much of it you'll likely be made a committer!
Someone needs to follow up on all the "Patch Available" in Jira to
decide if they're ready to be committed. Comments like, "+1 This
works for me, has tests, conforms to style, and all-in-all looks it
would make an improvement to the project", or "This can't be committed
as is because..." go a long ways towards getting issues resolved.
Even better is to post a version of the patch that can be committed.
Someone needs to test all the projects, running './build.sh clean test
dist' from the top-level. This often fails due to, e.g., problems in
licensing that have crept in that are identified by RAT.
Once all that's established then actually creating the release
candidate and calling the vote is pretty easy. This is best done by a
committer, since it involves creating SVN tags, updates to the CHANGES
I delivered a patch for AVRO-1352 (Schema corruption writing files in
C++) complete with test case, over three weeks ago and nobody has
followed up on it yet. I'd love to see this in 1.7.7.
Can someone please review and test it?
Senior Software Engineer | Lockheed Martin
On 26 Jun 2014, at 16:29, Doug Cutting <[EMAIL PROTECTED]> wrote:
Absolutely. A big part of the ASF's purpose is to help projects run smoothly without being totally dependent on any one person. So I don't mean to demand more of you, Doug -- you already do a huge amount. It was more a rallying cry for the wider community.
One difficulty I see is because Avro is such a diverse code base, consisting of various totally different languages, there are only a small number of people who are qualified to review a given patch. I try to keep an eye on JIRA and the mailing list for things I can review, but I know nothing about (say) C++ or Perl, so the best review I could do with those languages would be a rubber-stamp, which would do more harm than good.
So please, folks, if you understand what a patch is about, please help by commenting on it. There might not be may people besides you who understand it.
I'll have a play around with Docker, to see if I can create a shareable container image that has all the languages and tools installed for building and testing. If that works, it would make it easy for anyone to run the tests for all languages, and to create release candidates.
Ok, probably that change isn't worth the effort -- it would be more productive to improve the release automation, so that it's easier to release everything at once. I'll see what I can do in that regard.
On Fri, Jun 27, 2014 at 9:21 AM, Martin Kleppmann <[EMAIL PROTECTED]> wrote:
To help keep an eye on what needs reviewing I created a JIRA filter
for the review queue here:
That would be great! The number of languages that Avro supports now
makes this even more valuable. I don't know if you've seen it, but
Jeff Hammerbacher wrote up some instructions for doing a release on
EC2 a few years ago:
You might use that as a starting point, or just update it with
whatever you come up with.