Hey guys,

What are the current plans for the 21 release?

The roadmap on the wiki [1] says "minor releases are made regularly,
every few months." The 21 branch was created 5 months ago, and the
release dates in jira for the various core projects have all passed. I
know some projects, eg HBase, would like to get users on more recent
bits. What are the next steps for rolling a release?

Thanks,
Eli

[1] http://wiki.apache.org/hadoop/Roadmap

On Tue, Feb 16, 2010 at 2:12 PM, Eli Collins <[EMAIL PROTECTED]> wrote:
> Hey guys,
>
> What are the current plans for the 21 release?
>
Doesn't seem like there is much interest going by the deafening silence Eli.

Did it come up last night at the HUG?

> The roadmap on the wiki [1] says "minor releases are made regularly,
> every few months." The 21 branch was created 5 months ago, and the
> release dates in jira for the various core projects have all passed. I
> know some projects, eg HBase, would like to get users on more recent
> bits. What are the next steps for rolling a release?
>

Taking a look at blockers -- assuming to make a release, we just need
to clear blockers and roll a candidate -- it seems like common/core
and hdfs have but a few but mapreduce has a bunch.  Too many.

Does anyone detect activity in the blocker issues?  I've not made a
review.  Perhaps someone already has?

St.Ack

On Thu, Feb 18, 2010 at 10:54 AM, Stack <[EMAIL PROTECTED]> wrote:
> On Tue, Feb 16, 2010 at 2:12 PM, Eli Collins <[EMAIL PROTECTED]> wrote:
>> Hey guys,
>>
>> What are the current plans for the 21 release?
>>
> Doesn't seem like there is much interest going by the deafening silence Eli.
>
> Did it come up last night at the HUG?
>

It did, Owen gave an update in his talk and there was some follow on
discussion. The question of whether it makes sense to rebase 21 on
trunk came up, since 21 has gotten really out of date since it was
branched. What do you and others think of rebasing the 21 branch on
trunk once security is feature complete next month, and then targeting
an initial release a month after that? The rationale being that 21
branch is old, has not been getting the backports it needs, and the
community should be releasing relatively recent bits. In the mean time
we can focus on 21/trunk blockers. It doesn't look like there are a
ton more trunk blockers than 21 blockers. I know people, HBase in
particular, have been waiting for a 21 release for a while so delaying
further is a pain, and we should consider rolling what we've got, the
hope would be that the 21 branch would be more widely useful if it has
more recent features and fixes.

>> The roadmap on the wiki [1] says "minor releases are made regularly,
>> every few months." The 21 branch was created 5 months ago, and the
>> release dates in jira for the various core projects have all passed. I
>> know some projects, eg HBase, would like to get users on more recent
>> bits. What are the next steps for rolling a release?
>>
>
> Taking a look at blockers -- assuming to make a release, we just need
> to clear blockers and roll a candidate -- it seems like common/core
> and hdfs have but a few but mapreduce has a bunch.  Too many.
>
> Does anyone detect activity in the blocker issues?  I've not made a
> review.  Perhaps someone already has?

A decent number of the MR ones are patch available and quite a few
others are documentation related, nothing too scary. The core/hdfs
ones are pretty doable. We should make another pass to confirm that
they're all actually blockers. I'm happy to start working on the
blockers if that will help push the release forward.

Thanks,
Eli

Eli Collins wrote:
> What are the current plans for the 21 release?

Personally, I'd rather re-branch 21 from trunk in early March.  I
believe Y!'s security changes will be feature-complete in trunk by then.
  Symlinks and end-to-end Avro should also hopefully be committed by then.

I'd then propose we roll a 0.21.0 (alpha) release one month later, in
early April.

Henceforth, I'd like to see the project:
  - set branch dates at 6-month intervals
  - roll alpha releases one month after branch dates
  - roll bugfix releases as needed thereafter

At the onset of each six-month period we'd also need to decide what sort
of release we'll make, major or minor.  (Minor releases allow new
features, but remove no deprecated APIs.  Only major releases are
permitted to remove previously deprecated APIs.  All pre-1.0 releases
are major.)

We'd also designate a release master for each release, to drive the
process: making the branches, rolling the artifacts, and calling the
votes, etc.  Do we have any release master candidate volunteers for 0.21?

Doug

On 2/18/10 11:29 AM, "Doug Cutting" <[EMAIL PROTECTED]> wrote:

> Eli Collins wrote:
>> What are the current plans for the 21 release?
>
> Personally, I'd rather re-branch 21 from trunk in early March.  I
> believe Y!'s security changes will be feature-complete in trunk by then.
>   Symlinks and end-to-end Avro should also hopefully be committed by then.
>
> I'd then propose we roll a 0.21.0 (alpha) release one month later, in
> early April.

+1

I have seen/heard about people using the 0.21 branch. Won't it disturbing to
them that a new 0.21 branch that they might sync to in the future will have
a whole bunch of new code/features that they were not testing with earlier?

I thought that there is some benefit in rolling a 0.22 sometime in the
future and then marking 0.21 as "not for production usage".

thanks,
dhruba

On Thu, Feb 18, 2010 at 11:29 AM, Doug Cutting <[EMAIL PROTECTED]> wrote:

> Eli Collins wrote:
>
>> What are the current plans for the 21 release?
>>
>
> Personally, I'd rather re-branch 21 from trunk in early March.  I believe
> Y!'s security changes will be feature-complete in trunk by then.  Symlinks
> and end-to-end Avro should also hopefully be committed by then.
>
> I'd then propose we roll a 0.21.0 (alpha) release one month later, in early
> April.
>
> Henceforth, I'd like to see the project:
>  - set branch dates at 6-month intervals
>  - roll alpha releases one month after branch dates
>  - roll bugfix releases as needed thereafter
>
> At the onset of each six-month period we'd also need to decide what sort of
> release we'll make, major or minor.  (Minor releases allow new features, but
> remove no deprecated APIs.  Only major releases are permitted to remove
> previously deprecated APIs.  All pre-1.0 releases are major.)
>
> We'd also designate a release master for each release, to drive the
> process: making the branches, rolling the artifacts, and calling the votes,
> etc.  Do we have any release master candidate volunteers for 0.21?
>
> Doug
>

--
Connect to me at http://www.facebook.com/dhruba

On Feb 18, 2010, at 11:29 AM, Doug Cutting wrote:

My presentation basically said that:
   * Yahoo is currently running Yahoo's 0.20.8 on all 25,000 of our  
nodes.
   * We have made substantial numbers of feature back ports from trunk  
into our branch. Including:
      * improved Capacity scheduler
      * run MapReduce tasks as the real user
   * Hadoop 0.21 isn't stable yet and still has 28 blockers.
   * Furthermore, there are missing back ports that have been fixed in  
trunk but not 0.21.
   * We have a very tight timeline for adding security, which we can't  
slip
      * feature complete in february
      * deploy to first integration cluster in april
      * completely deployed on all clusters in august
   * To reduce risk, we are back porting security in to a branch of  
Yahoo's 0.20 branch.
   * By the time we are ready for the next version after security, the  
current 0.21 will be too stale to be interesting
   * So Yahoo will probably skip straight from Yahoo 0.20 with  
security to 0.22 in 2011.
   * Someone outside of Yahoo needs to step up and start addressing  
the blockers to get 0.21 released.

> Personally, I'd rather re-branch 21 from trunk in early March.

Rather than set a date, I think it is time to move to feature-based  
releases. We'd need to vote on the feature set, but looking for  
security, end-to-end avro, and symlinks seems like a reasonable list.  
That will avoid the large rush of commits the last week of the  
deadline, which has been counter productive.

-- Owen

>
>  * Someone outside of Yahoo needs to step up and start addressing the
> blockers to get 0.21 released.
>

Thanks for the update on the state of the world at Yahoo!, Owen--very
helpful. I think the community will step up to knock down some of the
blockers once we resolve what should be in the 0.21 release: the current
branch, or a rebase on trunk. Do you/Yahoo! have a preference on that front?

Rather than set a date, I think it is time to move to feature-based
> releases. We'd need to vote on the feature set, but looking for security,
> end-to-end avro, and symlinks seems like a reasonable list. That will avoid
> the large rush of commits the last week of the deadline, which has been
> counter productive.
>

Could someone give an example of a successful open source project that
follows a feature-based release cycle? From the research I've done, the
regular drumbeat of time-based releases seem to be more conducive to project
health. Always interested to hear otherwise. The example Owen cites of "the
large rush of commits the last week of the deadline" is certainly a good
argument in favor of feature-based releases; I'm curious to hear more.

Thanks,
Jeff

On Feb 18, 2010, at 11:28 AM, Eli Collins wrote:

> On Thu, Feb 18, 2010 at 10:54 AM, Stack <[EMAIL PROTECTED]> wrote:
>> On Tue, Feb 16, 2010 at 2:12 PM, Eli Collins <[EMAIL PROTECTED]>  
>> wrote:
>>> Hey guys,
>>>
>>> What are the current plans for the 21 release?
>>>
>> Doesn't seem like there is much interest going by the deafening  
>> silence Eli.
>>
>> Did it come up last night at the HUG?
>>
>
> It did, Owen gave an update in his talk and there was some follow on
> discussion. The question of whether it makes sense to rebase 21 on
> trunk came up, since 21 has gotten really out of date since it was
> branched. What do you and others think of rebasing the 21 branch on
> trunk once security is feature complete next month, and then targeting
> an initial release a month after that? The rationale being that 21
> branch is old, has not been getting the backports it needs, and the
> community should be releasing relatively recent bits. In the mean time
> we can focus on 21/trunk blockers. It doesn't look like there are a
> ton more trunk blockers than 21 blockers. I know people, HBase in
> particular, have been waiting for a 21 release for a while so delaying
> further is a pain, and we should consider rolling what we've got, the
> hope would be that the 21 branch would be more widely useful if it has
> more recent features and fixes.

If the community want the append features fast then we  are better off  
using the original 21 branch.
Basing a 21 off trunk will mean that there are more features with a  
new set of blockers and  and hence will take longer to fix.
sanjay

Owen O'Malley wrote:
> Rather than set a date, I think it is time to move to feature-based
> releases.

The problem with feature-based releases is that one feature can delay
releases arbitrarily, so a single feature can hold all others hostage.
Rather I think we need to be willing to remove or disable a feature in a
branch if it proves problematic.  If an addition causes regressions that
are not resolved promptly then that addition might be removed or
disabled.  This levels the playing field: features that doesn't cause
regressions can be made available in a release in a timely, predictable
manner and are not held back by more problematic features.

Doug

On Feb 18, 2010, at 3:06 PM, Jeff Hammerbacher wrote:

> I think the community will step up to knock down some of the
> blockers once we resolve what should be in the 0.21 release: the  
> current
> branch, or a rebase on trunk. Do you/Yahoo! have a preference on  
> that front?

Yahoo doesn't care. Even if we rebase the 0.21 branch, because of the  
timing, Yahoo will probably never deploy it. (It take months to push a  
release through QA and production testing, as I wrote the security  
release will hit the pipeline this year (code complete in february,  
first integration cluster in april, on all production clusters by  
august). Yahoo can't handle another big release until january 2011.

Personally, I'd prefer to rebase 0.21, especially after we have the  
Maven story straightened out. Generating good poms would be a huge win  
for downstream projects.

One big concern is that backwards incompatibility is a big cost.  
Especially if 0.21 (like 0.19) never gets wide deployment, I'd like to  
start a vote that we don't make any API incompatible in 0.22 relative  
to 0.20.

-- Owen

Thanks Owen, that's useful information. It sounds like the API
incompatibility vote can be a separate issue.

Do we have consensus around rebasing on 0.21? Anyone already testing on 0.21
who would be upset if the current branch were to be retired?

On Thu, Feb 18, 2010 at 5:02 PM, Owen O'Malley <[EMAIL PROTECTED]> wrote:

> On Feb 18, 2010, at 3:06 PM, Jeff Hammerbacher wrote:
>
>  I think the community will step up to knock down some of the
>> blockers once we resolve what should be in the 0.21 release: the current
>> branch, or a rebase on trunk. Do you/Yahoo! have a preference on that
>> front?
>>
>
> Yahoo doesn't care. Even if we rebase the 0.21 branch, because of the
> timing, Yahoo will probably never deploy it. (It take months to push a
> release through QA and production testing, as I wrote the security release
> will hit the pipeline this year (code complete in february, first
> integration cluster in april, on all production clusters by august). Yahoo
> can't handle another big release until january 2011.
>
> Personally, I'd prefer to rebase 0.21, especially after we have the Maven
> story straightened out. Generating good poms would be a huge win for
> downstream projects.
>
> One big concern is that backwards incompatibility is a big cost. Especially
> if 0.21 (like 0.19) never gets wide deployment, I'd like to start a vote
> that we don't make any API incompatible in 0.22 relative to 0.20.
>
> -- Owen
>

On Thu, Feb 18, 2010 at 5:19 PM, Jeff Hammerbacher <[EMAIL PROTECTED]> wrote:
> Thanks Owen, that's useful information. It sounds like the API
> incompatibility vote can be a separate issue.
>
> Do we have consensus around rebasing on 0.21? Anyone already testing on 0.21
> who would be upset if the current branch were to be retired?
>

One question I have is what to do with the JIRA tickets - we probably
need some kind of bulk edit, right? Anything that's currently resolved
with fix version 22 needs to be changed over to fix version 21?

> On Thu, Feb 18, 2010 at 5:02 PM, Owen O'Malley <[EMAIL PROTECTED]> wrote:
>
>> On Feb 18, 2010, at 3:06 PM, Jeff Hammerbacher wrote:
>>
>>  I think the community will step up to knock down some of the
>>> blockers once we resolve what should be in the 0.21 release: the current
>>> branch, or a rebase on trunk. Do you/Yahoo! have a preference on that
>>> front?
>>>
>>
>> Yahoo doesn't care. Even if we rebase the 0.21 branch, because of the
>> timing, Yahoo will probably never deploy it. (It take months to push a
>> release through QA and production testing, as I wrote the security release
>> will hit the pipeline this year (code complete in february, first
>> integration cluster in april, on all production clusters by august). Yahoo
>> can't handle another big release until january 2011.
>>
>> Personally, I'd prefer to rebase 0.21, especially after we have the Maven
>> story straightened out. Generating good poms would be a huge win for
>> downstream projects.
>>
>> One big concern is that backwards incompatibility is a big cost. Especially
>> if 0.21 (like 0.19) never gets wide deployment, I'd like to start a vote
>> that we don't make any API incompatible in 0.22 relative to 0.20.
>>
>> -- Owen
>>
>

Todd Lipcon wrote:
> One question I have is what to do with the JIRA tickets - we probably
> need some kind of bulk edit, right? Anything that's currently resolved
> with fix version 22 needs to be changed over to fix version 21?

Yes, that should be done if we decide to rebase the 0.21 branch.

Doug

On 2/18/2010 5:19 PM, Jeff Hammerbacher wrote:
> Do we have consensus around rebasing on 0.21? Anyone already testing on 0.21
> who would be upset if the current branch were to be retired?

Rebasing 0.21 will further delay the release.
In current 0.21 branch there is some 28 blockers,
which will take a couple of weeks to fix.
The rebased 0.21 will add to this more issues, and therefore
more time. Based on the experience I had time to stabilize
the release is measured in months rather than weeks.

--Konstantin
>
> On Thu, Feb 18, 2010 at 5:02 PM, Owen O'Malley<[EMAIL PROTECTED]>  wrote:
>
>> >  On Feb 18, 2010, at 3:06 PM, Jeff Hammerbacher wrote:
>> >
>> >    I think the community will step up to knock down some of the
>>> >>  blockers once we resolve what should be in the 0.21 release: the current
>>> >>  branch, or a rebase on trunk. Do you/Yahoo! have a preference on that
>>> >>  front?
>>> >>
>> >
>> >  Yahoo doesn't care. Even if we rebase the 0.21 branch, because of the
>> >  timing, Yahoo will probably never deploy it. (It take months to push a
>> >  release through QA and production testing, as I wrote the security release
>> >  will hit the pipeline this year (code complete in february, first
>> >  integration cluster in april, on all production clusters by august). Yahoo
>> >  can't handle another big release until january 2011.
>> >
>> >  Personally, I'd prefer to rebase 0.21, especially after we have the Maven
>> >  story straightened out. Generating good poms would be a huge win for
>> >  downstream projects.
>> >
>> >  One big concern is that backwards incompatibility is a big cost. Especially
>> >  if 0.21 (like 0.19) never gets wide deployment, I'd like to start a vote
>> >  that we don't make any API incompatible in 0.22 relative to 0.20.
>> >
>> >  -- Owen
>> >

On Thu, Feb 18, 2010 at 6:08 PM, Konstantin Shvachko <[EMAIL PROTECTED]> wrote:
> On 2/18/2010 5:19 PM, Jeff Hammerbacher wrote:
>>
>> Do we have consensus around rebasing on 0.21? Anyone already testing on
>> 0.21
>> who would be upset if the current branch were to be retired?
>
> Rebasing 0.21 will further delay the release.
> In current 0.21 branch there is some 28 blockers,
> which will take a couple of weeks to fix.
> The rebased 0.21 will add to this more issues, and therefore
> more time. Based on the experience I had time to stabilize
> the release is measured in months rather than weeks.

I agree that we're probably talking months rather than weeks. However,
I see a lot of the stabilization time as a fixed cost regardless of
the number of changes. Certainly there is an O(n) component too, but I
don't think the stabilization time of a rebased 21 is double the
stabilization time of current 21. Maybe more like 30% more? Do you
disagree?

-Todd
> --Konstantin
>
>
>>
>> On Thu, Feb 18, 2010 at 5:02 PM, Owen O'Malley<[EMAIL PROTECTED]>  wrote:
>>
>>> >  On Feb 18, 2010, at 3:06 PM, Jeff Hammerbacher wrote:
>>> >
>>> >    I think the community will step up to knock down some of the
>>>>
>>>> >>  blockers once we resolve what should be in the 0.21 release: the
>>>> >> current
>>>> >>  branch, or a rebase on trunk. Do you/Yahoo! have a preference on
>>>> >> that
>>>> >>  front?
>>>> >>
>>>
>>> >
>>> >  Yahoo doesn't care. Even if we rebase the 0.21 branch, because of the
>>> >  timing, Yahoo will probably never deploy it. (It take months to push a
>>> >  release through QA and production testing, as I wrote the security
>>> > release
>>> >  will hit the pipeline this year (code complete in february, first
>>> >  integration cluster in april, on all production clusters by august).
>>> > Yahoo
>>> >  can't handle another big release until january 2011.
>>> >
>>> >  Personally, I'd prefer to rebase 0.21, especially after we have the
>>> > Maven
>>> >  story straightened out. Generating good poms would be a huge win for
>>> >  downstream projects.
>>> >
>>> >  One big concern is that backwards incompatibility is a big cost.
>>> > Especially
>>> >  if 0.21 (like 0.19) never gets wide deployment, I'd like to start a
>>> > vote
>>> >  that we don't make any API incompatible in 0.22 relative to 0.20.
>>> >
>>> >  -- Owen
>>> >
>
>

If we include symlinks, security and Avro in 0.21, then what's the feature
set for 0.22? Do we have any big items planned?

Thanks,
Tomer

On Thu, Feb 18, 2010 at 6:12 PM, Todd Lipcon <[EMAIL PROTECTED]> wrote:

> On Thu, Feb 18, 2010 at 6:08 PM, Konstantin Shvachko <[EMAIL PROTECTED]>
> wrote:
> > On 2/18/2010 5:19 PM, Jeff Hammerbacher wrote:
> >>
> >> Do we have consensus around rebasing on 0.21? Anyone already testing on
> >> 0.21
> >> who would be upset if the current branch were to be retired?
> >
> > Rebasing 0.21 will further delay the release.
> > In current 0.21 branch there is some 28 blockers,
> > which will take a couple of weeks to fix.
> > The rebased 0.21 will add to this more issues, and therefore
> > more time. Based on the experience I had time to stabilize
> > the release is measured in months rather than weeks.
>
> I agree that we're probably talking months rather than weeks. However,
> I see a lot of the stabilization time as a fixed cost regardless of
> the number of changes. Certainly there is an O(n) component too, but I
> don't think the stabilization time of a rebased 21 is double the
> stabilization time of current 21. Maybe more like 30% more? Do you
> disagree?
>
> -Todd
>
>
> > --Konstantin
> >
> >
> >>
> >> On Thu, Feb 18, 2010 at 5:02 PM, Owen O'Malley<[EMAIL PROTECTED]>
>  wrote:
> >>
> >>> >  On Feb 18, 2010, at 3:06 PM, Jeff Hammerbacher wrote:
> >>> >
> >>> >    I think the community will step up to knock down some of the
> >>>>
> >>>> >>  blockers once we resolve what should be in the 0.21 release: the
> >>>> >> current
> >>>> >>  branch, or a rebase on trunk. Do you/Yahoo! have a preference on
> >>>> >> that
> >>>> >>  front?
> >>>> >>
> >>>
> >>> >
> >>> >  Yahoo doesn't care. Even if we rebase the 0.21 branch, because of
> the
> >>> >  timing, Yahoo will probably never deploy it. (It take months to push
> a
> >>> >  release through QA and production testing, as I wrote the security
> >>> > release
> >>> >  will hit the pipeline this year (code complete in february, first
> >>> >  integration cluster in april, on all production clusters by august).
> >>> > Yahoo
> >>> >  can't handle another big release until january 2011.
> >>> >
> >>> >  Personally, I'd prefer to rebase 0.21, especially after we have the
> >>> > Maven
> >>> >  story straightened out. Generating good poms would be a huge win for
> >>> >  downstream projects.
> >>> >
> >>> >  One big concern is that backwards incompatibility is a big cost.
> >>> > Especially
> >>> >  if 0.21 (like 0.19) never gets wide deployment, I'd like to start a
> >>> > vote
> >>> >  that we don't make any API incompatible in 0.22 relative to 0.20.
> >>> >
> >>> >  -- Owen
> >>> >
> >
> >
>

--
Tomer Shiran
Director of Product Management | MapR Technologies (www.mapr.com) |
650-804-8657

On Thu, Feb 18, 2010 at 5:02 PM, Owen O'Malley <[EMAIL PROTECTED]> wrote:
> Yahoo doesn't care. Even if we rebase the 0.21 branch, because of the
> timing, Yahoo will probably never deploy it. (It take months to push a
> release through QA and production testing, as I wrote the security release
> will hit the pipeline this year (code complete in february, first
> integration cluster in april, on all production clusters by august). Yahoo
> can't handle another big release until january 2011.

I haven't done a survey but my guess is that the HBase crew would
favor putting out the current 0.21 branch as 0.21 rather than
rebasing.  To us, hdfs and its flush, in testing, runs great.  I'm
with Konstantin and Sanjay that a release is pretty close and that a
rebasing pulling in new features means more months even allowing for
the Todd stabilizing constant.

If the project is not getting any heavy-duty Y!-fu till ~January 2011,
that sounds like hadoop 0.23 time, rather than 0.22, to me, that is if
we want to get the project back on 6-month cycles again (> 1 year
between major releases ain't healthy).

St.Ack

On Thu, Feb 18, 2010 at 8:36 PM, Stack <[EMAIL PROTECTED]> wrote:
> On Thu, Feb 18, 2010 at 5:02 PM, Owen O'Malley <[EMAIL PROTECTED]> wrote:
>> Yahoo doesn't care. Even if we rebase the 0.21 branch, because of the
>> timing, Yahoo will probably never deploy it. (It take months to push a
>> release through QA and production testing, as I wrote the security release
>> will hit the pipeline this year (code complete in february, first
>> integration cluster in april, on all production clusters by august). Yahoo
>> can't handle another big release until january 2011.
>
> I haven't done a survey but my guess is that the HBase crew would
> favor putting out the current 0.21 branch as 0.21 rather than
> rebasing.  To us, hdfs and its flush, in testing, runs great.  I'm
> with Konstantin and Sanjay that a release is pretty close and that a
> rebasing pulling in new features means more months even allowing for
> the Todd stabilizing constant.

Given that Yahoo and others plan to skip 21, and the HBase crowd could
actually use 21 and have invested in the current branch's contents, it
sounds like we should try to get the current branch released. Anyone
object?

Thanks,
Eli

+1 from me.

I agree with Stack; faster release cycles will help keep the project focused
and get code testing and soaking in more environments.

- Aaron

On Fri, Feb 19, 2010 at 1:44 PM, Eli Collins <[EMAIL PROTECTED]> wrote:

> On Thu, Feb 18, 2010 at 8:36 PM, Stack <[EMAIL PROTECTED]> wrote:
> > On Thu, Feb 18, 2010 at 5:02 PM, Owen O'Malley <[EMAIL PROTECTED]>
> wrote:
> >> Yahoo doesn't care. Even if we rebase the 0.21 branch, because of the
> >> timing, Yahoo will probably never deploy it. (It take months to push a
> >> release through QA and production testing, as I wrote the security
> release
> >> will hit the pipeline this year (code complete in february, first
> >> integration cluster in april, on all production clusters by august).
> Yahoo
> >> can't handle another big release until january 2011.
> >
> > I haven't done a survey but my guess is that the HBase crew would
> > favor putting out the current 0.21 branch as 0.21 rather than
> > rebasing.  To us, hdfs and its flush, in testing, runs great.  I'm
> > with Konstantin and Sanjay that a release is pretty close and that a
> > rebasing pulling in new features means more months even allowing for
> > the Todd stabilizing constant.
>
> Given that Yahoo and others plan to skip 21, and the HBase crowd could
> actually use 21 and have invested in the current branch's contents, it
> sounds like we should try to get the current branch released. Anyone
> object?
>
> Thanks,
> Eli
>

Eli Collins wrote:
> Given that Yahoo and others plan to skip 21, and the HBase crowd could
> actually use 21 and have invested in the current branch's contents, it
> sounds like we should try to get the current branch released.

My concern is more about when the next branch from trunk will be made.
If we embrace a six-month trunk branch cycle, then the next branch from
trunk should be created soon, regardless of whether we call it 21 or 22.

Also note that this decision has compatibility implications, unless we
decide to, post-fact, declare that 0.20 was actually a major release
(1.0) and that 0.21 (1.1) and 0.22 (1.2) are minor, feature releases,
that may add deprecations and features but may not remove any or
otherwise change APIs incompatibly.

Doug

On Fri, Feb 19, 2010 at 1:58 PM, Doug Cutting <[EMAIL PROTECTED]> wrote:
> Eli Collins wrote:
>>
>> Given that Yahoo and others plan to skip 21, and the HBase crowd could
>> actually use 21 and have invested in the current branch's contents, it
>> sounds like we should try to get the current branch released.
>
> My concern is more about when the next branch from trunk will be made. If we
> embrace a six-month trunk branch cycle, then the next branch from trunk
> should be created soon, regardless of whether we call it 21 or 22.
>

Does there need to be a dependency? We could still branch 22 once
security, avro, etc are in, even if that's only a couple months from
now. Minor releases are supposed to come out in a matter of months
anyway, since we haven't released a major version yet.

> Also note that this decision has compatibility implications, unless we
> decide to, post-fact, declare that 0.20 was actually a major release (1.0)
> and that 0.21 (1.1) and 0.22 (1.2) are minor, feature releases, that may add
> deprecations and features but may not remove any or otherwise change APIs
> incompatibly.

Could the vote on whether 22 is backwards compatible with 20 be
independent of what we call 1.0? Guess it depends on what type of
backwards compatibility we're talking about (eg API vs wire). Given
that already branched 21 a while back, it seems like we should be able
to release it, regardless of how a compatibility vote goes.

Thanks,
Eli

Eli Collins wrote:
>> My concern is more about when the next branch from trunk will be made.
>
> Does there need to be a dependency?

No.  I just wanted to note that, from my point of view, releasing from
the existing 0.21 branch is not sufficient, that, regardless, we still
need to release from trunk soon, and we need a schedule going forward
for when future branches from trunk will be made.

Aside from resolving compatibility issues (more on that below) what we
perhaps need are one or two release master volunteers, for a release
based on trunk and perhaps for a release based on the 0.21 branch.

> Could the vote on whether 22 is backwards compatible with 20 be
> independent of what we call 1.0?

We minimally need to declare whether releases are major or minor.  The
convention has been that all 0.X releases are major, 1.0 is major, 1.1,
1.2, etc. are minor, 2.0 is major, 2.1, 2.2, etc are minor and so on.
We could amend this convention, but things might get confusing.  For
example, we could declare that we just have a sequence of numerically
increasing releases, some major, some minor, but that you can't tell
which is which from the number.  Or we could, like Sun did with Java,
move the decimal point, and say that 0.20 is just 2.0, and that, instead
of 0.21 our next release is 2.1.  Or we could, as I suggested in my
previous message, retroactively consider 0.20 to be 1.0, and then make
our next release 1.1.  None are particularly attractive options.

Doug

Hadoop 2 EE Version 5?

On Fri, Feb 19, 2010 at 5:49 PM, Doug Cutting <[EMAIL PROTECTED]> wrote:

> Eli Collins wrote:
>
>> My concern is more about when the next branch from trunk will be made.
>>>
>>
>> Does there need to be a dependency?
>>
>
> No.  I just wanted to note that, from my point of view, releasing from the
> existing 0.21 branch is not sufficient, that, regardless, we still need to
> release from trunk soon, and we need a schedule going forward for when
> future branches from trunk will be made.
>
> Aside from resolving compatibility issues (more on that below) what we
> perhaps need are one or two release master volunteers, for a release based
> on trunk and perhaps for a release based on the 0.21 branch.
>
>
>  Could the vote on whether 22 is backwards compatible with 20 be
>> independent of what we call 1.0?
>>
>
> We minimally need to declare whether releases are major or minor.  The
> convention has been that all 0.X releases are major, 1.0 is major, 1.1, 1.2,
> etc. are minor, 2.0 is major, 2.1, 2.2, etc are minor and so on. We could
> amend this convention, but things might get confusing.  For example, we
> could declare that we just have a sequence of numerically increasing
> releases, some major, some minor, but that you can't tell which is which
> from the number.  Or we could, like Sun did with Java, move the decimal
> point, and say that 0.20 is just 2.0, and that, instead of 0.21 our next
> release is 2.1.  Or we could, as I suggested in my previous message,
> retroactively consider 0.20 to be 1.0, and then make our next release 1.1.
>  None are particularly attractive options.
>
> Doug
>

On Fri, Feb 19, 2010 at 2:49 PM, Doug Cutting <[EMAIL PROTECTED]> wrote:
> Eli Collins wrote:
>>>
>>> My concern is more about when the next branch from trunk will be made.
>>
>> Does there need to be a dependency?
>
> No.  I just wanted to note that, from my point of view, releasing from the
> existing 0.21 branch is not sufficient, that, regardless, we still need to
> release from trunk soon, and we need a schedule going forward for when
> future branches from trunk will be made.
>

Agreed. Releasing a 21 won't settle the release process question or
tell us when the first major release is. Maybe we could get 21 behind
us and have a separate discussion covering those.

>> Could the vote on whether 22 is backwards compatible with 20 be
>> independent of what we call 1.0?
>
> We minimally need to declare whether releases are major or minor.

I was assuming 21 would be another minor release, didn't hear
otherwise when it was branched. I didn't interpret the compatibility
vote as a suggestion that we should retroactively consider 20 to be
the first major release, but rather a suggestion for voting that we
don't remove APIs in 22 that were deprecated in 20. Owen, please
correct me if I'm wrong.

Thanks,
Eli

Eli Collins wrote:
> Maybe we could get 21 behind
> us and have a separate discussion covering those.

I'm personally more interested in that discussion than in what's
currently in the 0.21 branch, but others, like Stack, may reasonably be
of the opposite persuasion.  So rather than putting one before the
other, can we pursue both in parallel?

> I was assuming 21 would be another minor release, didn't hear
> otherwise when it was branched.

We've never officially had a minor release.  All of our releases have
been either major or bugfix.  A minor release adds features but does not
remove any deprecated APIs.  So making 0.21 a minor release would be a
change in the rules.

Have no deprecations in fact been removed between the 0.20 and 0.21
branches?

Perhaps we can agree that, after 0.20, no deprecations will be removed
until 1.0, that all pre-1.0 releases after 0.20 are now minor instead of
major, that 0.20 was effectively 1.0 but we're not going to rename it,
and that 1.0 will effectively be 2.0, etc.  Could that work?

Doug

On 2/19/10 3:38 PM, "Doug Cutting" <[EMAIL PROTECTED]> wrote:
> Perhaps we can agree that, after 0.20, no deprecations will be removed
> until 1.0, that all pre-1.0 releases after 0.20 are now minor instead of
> major, that 0.20 was effectively 1.0 but we're not going to rename it,
> and that 1.0 will effectively be 2.0, etc.  Could that work?

FWIW: I'm still very much opposed to any 1.0 that isn't ABI-wire compatible
going forward.

On Fri, Feb 19, 2010 at 3:38 PM, Doug Cutting <[EMAIL PROTECTED]> wrote:
> Eli Collins wrote:
>>
>> Maybe we could get 21 behind
>> us and have a separate discussion covering those.
>
> I'm personally more interested in that discussion than in what's currently
> in the 0.21 branch, but others, like Stack, may reasonably be of the
> opposite persuasion.  So rather than putting one before the other, can we
> pursue both in parallel?
>

Don't want to hold up the release process, version, compatibility
discussion, or have it hold up work on the 21 release (most 21
blockers are probably trunk blockers so probably not much wasted
effort either way).

>> I was assuming 21 would be another minor release, didn't hear
>> otherwise when it was branched.
>
> We've never officially had a minor release.  All of our releases have been
> either major or bugfix.  A minor release adds features but does not remove
> any deprecated APIs.  So making 0.21 a minor release would be a change in
> the rules.
>

Good to know, the versioning scheme (major.minor.point) outlined on
the wiki makes it seem like there's only been minor and point
releases.

Can we make a decision on basing 21 on the current branch and if it's
decided that 22 can't remove stuff that was in 20 we'll go back and do
the necessary additions on 21 and trunk? Suspect that decision will
take a lot more back and forth, but needs to conclude before 21 is
released.

Thanks,
Eli

On Fri, Feb 19, 2010 at 4:23 PM, Eli Collins <[EMAIL PROTECTED]> wrote:
> Can we make a decision on basing 21 on the current branch and if it's
> decided that 22 can't remove stuff that was in 20 we'll go back and do
> the necessary additions on 21 and trunk? Suspect that decision will
> take a lot more back and forth, but needs to conclude before 21 is
> released.
>

Lets.

Regards 0.21/current-branch release, as has been suggested above,
first we need to figure the release master.  No release master, no
release.  If we have a release master, then I suggest we vote on
current branch being released as 0.21 as soon as the blockers are
cleared.

I don't think we need muddy the above vote with whether or not 0.21
maintains API combatibility with 0.20.  IMO, it must (because Y! want
to have the 0.20 API in place when January 2011 rolls around).  This
makes 0.21 a "minor" release -- something we've not done before (For
the record, I also had a misunderstanding that what we were doing up
to this was major and patch only).  So, part of the release process
would involve ensuring no removed deprecations, etc.

As DC has been saying, this requirement that releases between now and
January 2011 not change APIs makes 0.20 retroactively into a "major"
release.  0.20 is the release where major shifted left in our
versioning scheme and minor releases came into play.  0.21 and 0.22
will be minor releases. Can we just acknowledge this fact, that there
was a step at version 0.20, update the wiki around versioning -- its
currently wrong anyways as Elis' points out -- and just move on?
Going back and calling 0.20 a 1.0 seems more apt to create confusion
and besides, I'm with Allen that hadoop 1.0 needs wire compatibility
before the 1.0 can roll around.

Thanks,
St.Ack
P.S. +1 on branching as soon as avro and security are in, etc.

Well, since someone has to get the ball rolling as far as release  
masters, I'll nominate Stack and/or someone hbase related for 0.21  
with the primary goal of being "soon"?  They get a big win from append  
and others will gain from the expanded mapreduce lib, better  
schedulers, etc. There are a lot of new features and some major  
changes (project split) already in the 0.21 branch, so IMO it's worth  
considering a release with minimal backports, rather than make binding  
decisions about 0.22 before 0.21 is even in the wild.

-Jay

PS sorry Stack

On Feb 20, 2010, at 5:04 PM, Stack <[EMAIL PROTECTED]> wrote:

> On Fri, Feb 19, 2010 at 4:23 PM, Eli Collins <[EMAIL PROTECTED]> wrote:
>> Can we make a decision on basing 21 on the current branch and if it's
>> decided that 22 can't remove stuff that was in 20 we'll go back and  
>> do
>> the necessary additions on 21 and trunk? Suspect that decision will
>> take a lot more back and forth, but needs to conclude before 21 is
>> released.
>>
>
> Lets.
>
> Regards 0.21/current-branch release, as has been suggested above,
> first we need to figure the release master.  No release master, no
> release.  If we have a release master, then I suggest we vote on
> current branch being released as 0.21 as soon as the blockers are
> cleared.
>
> I don't think we need muddy the above vote with whether or not 0.21
> maintains API combatibility with 0.20.  IMO, it must (because Y! want
> to have the 0.20 API in place when January 2011 rolls around).  This
> makes 0.21 a "minor" release -- something we've not done before (For
> the record, I also had a misunderstanding that what we were doing up
> to this was major and patch only).  So, part of the release process
> would involve ensuring no removed deprecations, etc.
>
> As DC has been saying, this requirement that releases between now and
> January 2011 not change APIs makes 0.20 retroactively into a "major"
> release.  0.20 is the release where major shifted left in our
> versioning scheme and minor releases came into play.  0.21 and 0.22
> will be minor releases. Can we just acknowledge this fact, that there
> was a step at version 0.20, update the wiki around versioning -- its
> currently wrong anyways as Elis' points out -- and just move on?
> Going back and calling 0.20 a 1.0 seems more apt to create confusion
> and besides, I'm with Allen that hadoop 1.0 needs wire compatibility
> before the 1.0 can roll around.
>
> Thanks,
> St.Ack
> P.S. +1 on branching as soon as avro and security are in, etc.

I'm sorry for breaking into your discussion as an outsider, but I'm very
curious about the security features you are planning to roll out in March.
Where can I find information about this?

Best regards,
Evert Lammerts

> -----Original Message-----
> From: Jay Booth [mailto:[EMAIL PROTECTED]]
> Sent: maandag 22 februari 2010 5:55
> To: [EMAIL PROTECTED]
> Cc: [EMAIL PROTECTED]
> Subject: Re: Release plans
>
> Well, since someone has to get the ball rolling as far as release
> masters, I'll nominate Stack and/or someone hbase related for 0.21
> with the primary goal of being "soon"?  They get a big win from append
> and others will gain from the expanded mapreduce lib, better
> schedulers, etc. There are a lot of new features and some major
> changes (project split) already in the 0.21 branch, so IMO it's worth
> considering a release with minimal backports, rather than make binding
> decisions about 0.22 before 0.21 is even in the wild.
>
> -Jay
>
> PS sorry Stack
>
> On Feb 20, 2010, at 5:04 PM, Stack <[EMAIL PROTECTED]> wrote:
>
> > On Fri, Feb 19, 2010 at 4:23 PM, Eli Collins <[EMAIL PROTECTED]>
> wrote:
> >> Can we make a decision on basing 21 on the current branch and if
> it's
> >> decided that 22 can't remove stuff that was in 20 we'll go back and
> >> do
> >> the necessary additions on 21 and trunk? Suspect that decision will
> >> take a lot more back and forth, but needs to conclude before 21 is
> >> released.
> >>
> >
> > Lets.
> >
> > Regards 0.21/current-branch release, as has been suggested above,
> > first we need to figure the release master.  No release master, no
> > release.  If we have a release master, then I suggest we vote on
> > current branch being released as 0.21 as soon as the blockers are
> > cleared.
> >
> > I don't think we need muddy the above vote with whether or not 0.21
> > maintains API combatibility with 0.20.  IMO, it must (because Y! want
> > to have the 0.20 API in place when January 2011 rolls around).  This
> > makes 0.21 a "minor" release -- something we've not done before (For
> > the record, I also had a misunderstanding that what we were doing up
> > to this was major and patch only).  So, part of the release process
> > would involve ensuring no removed deprecations, etc.
> >
> > As DC has been saying, this requirement that releases between now and
> > January 2011 not change APIs makes 0.20 retroactively into a "major"
> > release.  0.20 is the release where major shifted left in our
> > versioning scheme and minor releases came into play.  0.21 and 0.22
> > will be minor releases. Can we just acknowledge this fact, that there
> > was a step at version 0.20, update the wiki around versioning -- its
> > currently wrong anyways as Elis' points out -- and just move on?
> > Going back and calling 0.20 a 1.0 seems more apt to create confusion
> > and besides, I'm with Allen that hadoop 1.0 needs wire compatibility
> > before the 1.0 can roll around.
> >
> > Thanks,
> > St.Ack
> > P.S. +1 on branching as soon as avro and security are in, etc.

On Feb 19, 2010, at 3:14 PM, Eli Collins wrote:

> On Fri, Feb 19, 2010 at 2:49 PM, Doug Cutting <[EMAIL PROTECTED]>  
> wrote:
>> Eli Collins wrote:
>>>>
>>>> My concern is more about when the next branch from trunk will be  
>>>> made.
>>>
>>> Does there need to be a dependency?
>>
>> No.  I just wanted to note that, from my point of view, releasing  
>> from the
>> existing 0.21 branch is not sufficient, that, regardless, we still  
>> need to
>> release from trunk soon, and we need a schedule going forward for  
>> when
>> future branches from trunk will be made.
>>
>
> Agreed. Releasing a 21 won't settle the release process question or
> tell us when the first major release is. Maybe we could get 21 behind
> us and have a separate discussion covering those.
>
>>> Could the vote on whether 22 is backwards compatible with 20 be
>>> independent of what we call 1.0?
>>
>> We minimally need to declare whether releases are major or minor.
>
> I was assuming 21 would be another minor release, didn't hear
> otherwise when it was branched. I didn't interpret the compatibility
> vote as a suggestion that we should retroactively consider 20 to be
> the first major release, but rather a suggestion for voting that we
> don't remove APIs in 22 that were deprecated in 20.
Agreed.
Rather than renaming versions, lets vote on the compatibility rules  
for 22: 22 is API compatible with 20.
> Owen, please
> correct me if I'm wrong.
>
> Thanks,
> Eli

Sanjay Radia wrote:
> Rather than renaming versions, lets vote on the compatibility rules for
> 22: 22 is API compatible with 20.

I think if we elect to go this way then we should make the rule more
general, i.e.: all releases after 0.20 are to be API-compatible with
0.20 until we state otherwise (probably 1.0).  If we both release 0.21
as currently branched and we branch 0.22 from trunk in a few weeks, then
we might want to have an API-compatible 0.23 too.

Doug