Hello everyone,

This has been informally mentioned before, but I think it's best to be
completely transparent/explicit about this.

We (Sanjay, Suresh, Todd, Eli, myself, and anyone else who wants to help)
intend to do the work for HDFS-1623 (High Availability Framework for HDFS
NN) on a development branch off of trunk. The work in the HDFS-1073
development branch is necessary to complete HDFS-1623. As such, we're
waiting for the work in HDFS-1073 to be merged into trunk before creating a
branch for HDFS-1623.

Once this branch is created, I'd like to use a similar modified
commit-then-review policy for this branch as was done in the HDFS-1073
branch, which I think worked very well. To review, this was:

{quote}
- A patch will be uploaded to the JIRA for review like usual
- If another committer provides a +1, it may be committed at that
point, just like usual.
- If no committer provides +1 (or a review asking for changes) within
24 business hours, it will be committed to the branch under "commit then
review" policy.Of course if any committer feels that code needs to be
amended, he or she should feel free to open a new JIRA against the branch
including the review comments, and they will be addressed before the merge
into trunk. And just like with any branch merge, ample time will be given
for the community to review both the large merge commit as well as the
individual historical commits of the branch, before it goes into trunk.
{quote}

I'm also volunteering to keep the HDFS-1623 development branch up to date
with respect to merging the concurrent changes which go into trunk into this
development branch to make sure the merge back into trunk is as painless as
possible.

Comments are certainly welcome on this strategy.

Thanks a lot,
Aaron

--
Aaron T. Myers
 Software Engineer, Cloudera

Sounds good to me. I think this strategy has worked well on the HDFS-1073
branch -- allowed development to be quite rapid, and at this point all but a
couple trivial patches have been explicitly reviewed by a committer (and the
others implicitly reviewed since later patches touched the same code area).

+1.

-Todd

On Thu, Jul 7, 2011 at 1:43 PM, Aaron T. Myers <[EMAIL PROTECTED]> wrote:

> Hello everyone,
>
> This has been informally mentioned before, but I think it's best to be
> completely transparent/explicit about this.
>
> We (Sanjay, Suresh, Todd, Eli, myself, and anyone else who wants to help)
> intend to do the work for HDFS-1623 (High Availability Framework for HDFS
> NN) on a development branch off of trunk. The work in the HDFS-1073
> development branch is necessary to complete HDFS-1623. As such, we're
> waiting for the work in HDFS-1073 to be merged into trunk before creating a
> branch for HDFS-1623.
>
> Once this branch is created, I'd like to use a similar modified
> commit-then-review policy for this branch as was done in the HDFS-1073
> branch, which I think worked very well. To review, this was:
>
> {quote}
> - A patch will be uploaded to the JIRA for review like usual
> - If another committer provides a +1, it may be committed at that
> point, just like usual.
> - If no committer provides +1 (or a review asking for changes) within
> 24 business hours, it will be committed to the branch under "commit then
> review" policy.Of course if any committer feels that code needs to be
> amended, he or she should feel free to open a new JIRA against the branch
> including the review comments, and they will be addressed before the merge
> into trunk. And just like with any branch merge, ample time will be given
> for the community to review both the large merge commit as well as the
> individual historical commits of the branch, before it goes into trunk.
> {quote}
>
> I'm also volunteering to keep the HDFS-1623 development branch up to date
> with respect to merging the concurrent changes which go into trunk into
> this
> development branch to make sure the merge back into trunk is as painless as
> possible.
>
> Comments are certainly welcome on this strategy.
>
> Thanks a lot,
> Aaron
>
> --
> Aaron T. Myers
>  Software Engineer, Cloudera
>

--
Todd Lipcon
Software Engineer, Cloudera

+1

Sounds like this has worked well for the MR2 branch as well.

Thanks for volunteering to maintain the branch.

On Thu, Jul 7, 2011 at 1:43 PM, Aaron T. Myers <[EMAIL PROTECTED]> wrote:
> Hello everyone,
>
> This has been informally mentioned before, but I think it's best to be
> completely transparent/explicit about this.
>
> We (Sanjay, Suresh, Todd, Eli, myself, and anyone else who wants to help)
> intend to do the work for HDFS-1623 (High Availability Framework for HDFS
> NN) on a development branch off of trunk. The work in the HDFS-1073
> development branch is necessary to complete HDFS-1623. As such, we're
> waiting for the work in HDFS-1073 to be merged into trunk before creating a
> branch for HDFS-1623.
>
> Once this branch is created, I'd like to use a similar modified
> commit-then-review policy for this branch as was done in the HDFS-1073
> branch, which I think worked very well. To review, this was:
>
> {quote}
> - A patch will be uploaded to the JIRA for review like usual
> - If another committer provides a +1, it may be committed at that
> point, just like usual.
> - If no committer provides +1 (or a review asking for changes) within
> 24 business hours, it will be committed to the branch under "commit then
> review" policy.Of course if any committer feels that code needs to be
> amended, he or she should feel free to open a new JIRA against the branch
> including the review comments, and they will be addressed before the merge
> into trunk. And just like with any branch merge, ample time will be given
> for the community to review both the large merge commit as well as the
> individual historical commits of the branch, before it goes into trunk.
> {quote}
>
> I'm also volunteering to keep the HDFS-1623 development branch up to date
> with respect to merging the concurrent changes which go into trunk into this
> development branch to make sure the merge back into trunk is as painless as
> possible.
>
> Comments are certainly welcome on this strategy.
>
> Thanks a lot,
> Aaron
>
> --
> Aaron T. Myers
>  Software Engineer, Cloudera
>

+1, this sounds like a very good approach.

-dhruba

On Thu, Jul 7, 2011 at 4:33 PM, Eli Collins <[EMAIL PROTECTED]> wrote:

> +1
>
> Sounds like this has worked well for the MR2 branch as well.
>
> Thanks for volunteering to maintain the branch.
>
> On Thu, Jul 7, 2011 at 1:43 PM, Aaron T. Myers <[EMAIL PROTECTED]> wrote:
> > Hello everyone,
> >
> > This has been informally mentioned before, but I think it's best to be
> > completely transparent/explicit about this.
> >
> > We (Sanjay, Suresh, Todd, Eli, myself, and anyone else who wants to help)
> > intend to do the work for HDFS-1623 (High Availability Framework for HDFS
> > NN) on a development branch off of trunk. The work in the HDFS-1073
> > development branch is necessary to complete HDFS-1623. As such, we're
> > waiting for the work in HDFS-1073 to be merged into trunk before creating
> a
> > branch for HDFS-1623.
> >
> > Once this branch is created, I'd like to use a similar modified
> > commit-then-review policy for this branch as was done in the HDFS-1073
> > branch, which I think worked very well. To review, this was:
> >
> > {quote}
> > - A patch will be uploaded to the JIRA for review like usual
> > - If another committer provides a +1, it may be committed at that
> > point, just like usual.
> > - If no committer provides +1 (or a review asking for changes) within
> > 24 business hours, it will be committed to the branch under "commit then
> > review" policy.Of course if any committer feels that code needs to be
> > amended, he or she should feel free to open a new JIRA against the branch
> > including the review comments, and they will be addressed before the
> merge
> > into trunk. And just like with any branch merge, ample time will be given
> > for the community to review both the large merge commit as well as the
> > individual historical commits of the branch, before it goes into trunk.
> > {quote}
> >
> > I'm also volunteering to keep the HDFS-1623 development branch up to date
> > with respect to merging the concurrent changes which go into trunk into
> this
> > development branch to make sure the merge back into trunk is as painless
> as
> > possible.
> >
> > Comments are certainly welcome on this strategy.
> >
> > Thanks a lot,
> > Aaron
> >
> > --
> > Aaron T. Myers
> >  Software Engineer, Cloudera
> >
>

--
Connect to me at http://www.facebook.com/dhruba

Thanks Aaron for sending the details from the meeting that Sanjay
had arranged. I am very happy that you have
agreed to collaborate on our HA design published in HDFS-1623.

This is a critical feature for HDFS. This proposal is not exactly what I had
envisioned. Why don’t we meet again to
discuss and come up with a proposal for branching and development-process?

If there are other folks who want to contribute to development of HDFS-1623,
please reach out to me and I will send
the meeting details.

On Jul 8, 2011, at 12:22 PM, Suresh Srinivas wrote:

> Thanks Aaron for sending the details from the meeting that Sanjay
> had arranged.
What meeting?

On Jul 8, 2011, at 12:52 PM, Allen Wittenauer wrote:

>
> On Jul 8, 2011, at 12:22 PM, Suresh Srinivas wrote:
>
>> Thanks Aaron for sending the details from the meeting that Sanjay
>> had arranged.
>
>
> What meeting?
>
>
After we posted the design document, some HDFS contributors who participated in the Jira  reached out
to have a high bandwidth discussion to get clarification on the design document in HDFS 1623.
Comments  posted in the jira captured the feedback on the document.
An updated version of the document will incorporate the feedback.
Participants were suresh, kan, sanjay, jitendra, eli, todd, aaron m, dhruba, dmytro.
The agenda for the last contributors's meeting included a presentation of the HA design; unfortunately we ran out of time. I have requested a slot to present the design at the next HUG.
sanjay

On Jul 8, 2011, at 2:03 PM, sanjay Radia wrote:
>
>
> After we posted the design document, some HDFS contributors who participated in the Jira  reached out
> to have a high bandwidth discussion to get clarification on the design document in HDFS 1623.

  Was the public invited to this meeting?  Did I just miss the invite on the mailing lists when I went to search for it?

On 08/07/11 22:03, sanjay Radia wrote:
>
> On Jul 8, 2011, at 12:52 PM, Allen Wittenauer wrote:
>
>>
>> On Jul 8, 2011, at 12:22 PM, Suresh Srinivas wrote:
>>
>>> Thanks Aaron for sending the details from the meeting that Sanjay
>>> had arranged.
>>
>>
>> What meeting?
>>
>>
>
>
> After we posted the design document, some HDFS contributors who participated in the Jira  reached out
> to have a high bandwidth discussion to get clarification on the design document in HDFS 1623.
> Comments  posted in the jira captured the feedback on the document.
> An updated version of the document will incorporate the feedback.
> Participants were suresh, kan, sanjay, jitendra, eli, todd, aaron m, dhruba, dmytro.
>
>
> The agenda for the last contributors's meeting included a presentation of the HA design; unfortunately we ran out of time. I have requested a slot to present the design at the next HUG.

It'd be nice if some of these discussions were held online, so that
those of us who don't live in the Bay Area can take part. Otherwise
there's the risk of partitioning the community, and the CAP theorem says
that's bad news. We end up inconsistent or not willing to participate in
the project (i.e. unavailable)

It also leads to situations that I've seen in other ASF projects where
one large organisation would make decisions, execute and announce before
anyone else knew what was going on.

I can offer some shared desktop tools to share slideware (works on
windows, Mac and Linux), but the voice side of that tends to go to
voice-conf. If we could use skype or google + conf calling (or similar),
that would give us the voice.

We could then hold intermittent RT meetings. There are still TZ issues;
in past standards work the conf time rolled between times that worked
well for EU, US and asia, such that two continents at a time could
participate

Sanjay,

1. I don't see any comments in HDFS-1623 since may 26.
And I don't see any updates to the design document.
While HA jiras are being committed directly to trunk.
And I personally don't like what is being committed.
Possible because I don't understand the approach.

2. Whatever the branching strategy is, the branch should be created
and the work should be done in the branch.
Please call a vote to create the branch whoever is working on it.
I don't see anybody is opposing. My +1 in advance.

3. CTR or RTC I am seriously considering to revert HDFS-2141 because
it is changing the terminology used in several design documents ans
publications,
which people got used to, and which will create more confusion.

4. Sanjay's and Suresh's document is a great strategic design for HA.
Although I do not think you can use it to implement a single approach.
As it has a lot of different choices , which make the strategy great, but
need to be resolved for practical purposes.
An implementation requires (so to speak) a tactical design doc, which
clarifies the choices taken for the implementation.
I made an attempt to resolve the choices in HDFS-2064, but you seem to
disagree with them, although I don't know which ones.

I don't think any changes should go to trunk (or the branch) without such a
document.

Thanks,
--Konstantin

On Fri, Jul 8, 2011 at 2:03 PM, sanjay Radia <[EMAIL PROTECTED]> wrote:

>
> On Jul 8, 2011, at 12:52 PM, Allen Wittenauer wrote:
>
> >
> > On Jul 8, 2011, at 12:22 PM, Suresh Srinivas wrote:
> >
> >> Thanks Aaron for sending the details from the meeting that Sanjay
> >> had arranged.
> >
> >
> >       What meeting?
> >
> >
>
>
> After we posted the design document, some HDFS contributors who
> participated in the Jira  reached out
> to have a high bandwidth discussion to get clarification on the design
> document in HDFS 1623.
> Comments  posted in the jira captured the feedback on the document.
> An updated version of the document will incorporate the feedback.
> Participants were suresh, kan, sanjay, jitendra, eli, todd, aaron m,
> dhruba, dmytro.
>
>
> The agenda for the last contributors's meeting included a presentation of
> the HA design; unfortunately we ran out of time. I have requested a slot to
> present the design at the next HUG.
>
>
> sanjay

> CTR or RTC I am seriously considering to revert HDFS-2141 because
it is changing the terminology used in several design documents ans
publications,
which people got used to, and which will create more confusion.

Konstantin, you could comment this on jira. This is not a branching issue.
This is an issue you have with a specific jira!

BTW 2141 does clearly tell you why the previous naming is incorrect. Please
comment on the jira and lets continue the discussion there.

On 07/08/2011 12:22 PM, Suresh Srinivas wrote:
> This is a critical feature for HDFS. This proposal is not exactly
> what I had envisioned. Why don�t we meet again to discuss and come up
> with a proposal for branching and development-process?

Suresh,

What are your concerns with this proposal?

Thanks,

Doug

I also have concerns.  While those working on the 1073 branch haven't
abused the commit-then-review process, the fact remains that the final
merge to trunk is equivalent to a 1.3MB patch that makes far-reaching
changes to the guts of the NN.  Asking the reviewers most experienced
in this code to do such a herculean review such a merge is an
unreasonable request.  So (again with no malice on anyone's part),
we've ended up with a huge, far-reaching changeset that was committed
under a liberal policy that's not accepted on trunk being merged into
trunk based on a single +1 (per the bylaws as I read them, it's not
necessary to have more than that to merge a branch to trunk).  This is
not a good situation.

The tweak I would make to the suggested process is to require that the
final trunk merge have a higher threshold to commit.  Three +1s from
committers would be my suggestion.  This would also be an area where
cross-organization reviews would help significantly, but there's
nothing in the ASF to mandate such a requirement.

Finally, the private meeting that led to these discussions was
unfortunate, exclusionary and against the Apache Way.  Do we really
need another one to work this out?

-Jakob
On Fri, Jul 8, 2011 at 3:17 PM, Doug Cutting <[EMAIL PROTECTED]> wrote:
> On 07/08/2011 12:22 PM, Suresh Srinivas wrote:
>> This is a critical feature for HDFS. This proposal is not exactly
>> what I had envisioned. Why don’t we meet again to discuss and come up
>> with a proposal for branching and development-process?
>
> Suresh,
>
> What are your concerns with this proposal?
>
> Thanks,
>
> Doug
>

Hi Jakob,

My apologies for taking so long to get back to you - I've been traveling the
last few days.

On Fri, Jul 8, 2011 at 3:40 PM, Jakob Homan <[EMAIL PROTECTED]> wrote:

> The tweak I would make to the suggested process is to require that the
> final trunk merge have a higher threshold to commit.  Three +1s from
> committers would be my suggestion.
>

This seems like a good suggestion to me. The proposal I put forth is not at
all intended to diminish the quantity or quality of reviews, but rather to
make sure that the actual coding (commits to the branch, maintenance of the
branch, and collaboration on the branch) can progress without being stalled
waiting on reviews for each patch. The more reviewers the better, in my
opinion.

I would only ask that when the announcement is made that this branch is
ready for merge into trunk that those who intend to review it do so as
promptly as possible, and perhaps even announce their intention to do so, so
we can know who to wait on for reviews.

Do we need to vote on this change of policy?
> Finally, the private meeting that led to these discussions was
> unfortunate, exclusionary and against the Apache Way.
>

You're right - an invitation to the meeting should have been announced. That
said, I can assure you that there was nothing nefarious about this meeting.
Rather, the group of contributors who had previously expressed interest in
implementing NN HA met to review the proposal that was posted on HDFS-1623
to make sure everyone was on the same page with respect to the design.
Regardless of the innocence of this meeting, it still should have been
announced publicly.
> Do we really need another one to work this out?
>
>
I don't yet know what concerns Suresh has, but I'd be surprised if they
couldn't be worked out on the mailing lists.

--
Aaron T. Myers
Software Engineer, Cloudera

On Fri, Jul 8, 2011 at 3:40 PM, Jakob Homan <[EMAIL PROTECTED]> wrote:
> I also have concerns.  While those working on the 1073 branch haven't
> abused the commit-then-review process, the fact remains that the final
> merge to trunk is equivalent to a 1.3MB patch that makes far-reaching
> changes to the guts of the NN.

Each change was done individually with it's own jira, patch, and
review.  People can review the sub-tasks if they don't want to look at
the entire patch. The majority of the changes were reviewed before
they were committed, when that wasn't the case review feedback was
incorporated in follow-on patches. I don't see how this is any
different from eg how federation was developed.

> Asking the reviewers most experienced
> in this code to do such a herculean review such a merge is an
> unreasonable request.  So (again with no malice on anyone's part),
> we've ended up with a huge, far-reaching changeset that was committed
> under a liberal policy that's not accepted on trunk being merged into

Look at the sub-tasks on jira. This was not a liberal policy. All the
changes are done as individual jiras and with individual patches for
review.  The design doc was reviewed and discussed by a number of
people.

> trunk based on a single +1 (per the bylaws as I read them, it's not
> necessary to have more than that to merge a branch to trunk).  This is
> not a good situation.

The vast majority of changes that go into HDFS (and MR2 from my
understanding) are reviewed by a single person and others besides
myself (eg Sanjay, Ivan, and atm)  have reviewed patches on 1073
sub-tasks, so I don't see what the concern is. The new design, code,
and *huge* increase in test coverage is a massive improvement over
what we have currently.

I agree, in general, it would be great to see more reviewers, of both
the individual changes and the final merge.

Thanks,
Eli

Hi suresh,

Can you pl explain how we can modify Aaron's proposal to fit your needs? It
would be nice to do the HA development in its own branch, is it not?

-dhruba

On Fri, Jul 8, 2011 at 2:22 PM, Suresh Srinivas <[EMAIL PROTECTED]>wrote:

> Thanks Aaron for sending the details from the meeting that Sanjay
> had arranged. I am very happy that you have
> agreed to collaborate on our HA design published in HDFS-1623.
>
> This is a critical feature for HDFS. This proposal is not exactly what I
> had
> envisioned. Why don’t we meet again to
> discuss and come up with a proposal for branching and development-process?
>
> If there are other folks who want to contribute to development of
> HDFS-1623,
> please reach out to me and I will send
> the meeting details.
>

--
Connect to me at http://www.facebook.com/dhruba

Clearly we cannot get all the changes required for HA into 23. We would like to get some changes in that will
allow one to build a HA solution before a 23.x release.
What we want to discuss is what are the changes worth getting in into the current 0.23.
Further many of the changes are independent of each other and a single 1623 branch may not make sense.

sanjay
On Jul 8, 2011, at 12:22 PM, Suresh Srinivas wrote:

> Thanks Aaron for sending the details from the meeting that Sanjay
> had arranged. I am very happy that you have
> agreed to collaborate on our HA design published in HDFS-1623.
>
> This is a critical feature for HDFS. This proposal is not exactly what I had
> envisioned. Why don’t we meet again to
> discuss and come up with a proposal for branching and development-process?
>
> If there are other folks who want to contribute to development of HDFS-1623,
> please reach out to me and I will send
> the meeting details.

Sorry for the late reply...

My concern is not about creating a branch. Using a branch is some thing
we have been doing for a long time for feature development and it is
effective. Infact I am eager for branch for HA to be created, to start
submitting the patches I am currently working on.

Here are some of my concerns:
Commit-then-review:
I have same concerns expressed by Jakob, about commit and review. My
preference is to review before commit. Reviewing incremental changes is much
more effective than reviewing one mega patch during merge of a branch.

Diverging solutions:
Second unrelated problem to branching is - in HDFS-1623, the design defines
common elements to build HA solution. The idea was, it could enable
different
solutions depending on how folks want to go about it. I would like to make
sure that we are all converging, instead of diverging. This means some of
the
jiras still need to be discussed, especially the ones that make major
structural changes, adds interfaces etc. I am not sure 24 hr is a long
enough
notice.

>From what I also know, HDFS-2064 HA is being planned for release 0.22.
At an early glance, this seems to be closely tied to Backup node and not
generic enough (will post comments to jira). Given that this is added to
0.22,
how does that impact the current HA plans and backward compatibility
requirements?

Release 0.23:
Initial plan was to make HA part of post 0.23 (as discussed in contributors
meetup). It would be good to work in that direction and commit changes to
HDFS-1623 branch instead of making changes in the trunk, to ensure stability
of the trunk. HADOOP-7380 went into trunk. It would be better for such
changes
to go into HDFS-1623 branch. We should merge this branch post 0.23 into
trunk.

Regards,
Suresh

On Mon, Jul 11, 2011 at 1:28 PM, Suresh Srinivas <[EMAIL PROTECTED]> wrote:
> Sorry for the late reply...
>
> My concern is not about creating a branch. Using a branch is some thing
> we have been doing for a long time for feature development and it is
> effective. Infact I am eager for branch for HA to be created, to start
> submitting the patches I am currently working on.
>
> Here are some of my concerns:
> Commit-then-review:
> I have same concerns expressed by Jakob, about commit and review. My
> preference is to review before commit. Reviewing incremental changes is much
> more effective than reviewing one mega patch during merge of a branch.

In CTR you still review individual changes, it just means you can do
the review after the change was committed (and you address review
feedback in a follow-on patch).  I'm actually happy to do RTC, I
prefer that as well. As long as each change is individually reviewed
I'm fine either way.

> From what I also know, HDFS-2064 HA is being planned for release 0.22.

That's up to the 0.22 RM. It's not a blocker and it's unclear if we'll
have a 0.22 release.

> At an early glance, this seems to be closely tied to Backup node and not
> generic enough (will post comments to jira). Given that this is added to
> 0.22,
> how does that impact the current HA plans and backward compatibility
> requirements?

I don't think the 1632 work needs to gate 2064, and vice versa.

> Release 0.23:
> Initial plan was to make HA part of post 0.23 (as discussed in contributors
> meetup). It would be good to work in that direction and commit changes to
> HDFS-1623 branch instead of making changes in the trunk, to ensure stability
> of the trunk. HADOOP-7380 went into trunk. It would be better for such
> changes
> to go into HDFS-1623 branch. We should merge this branch post 0.23 into
> trunk.

Per the summit and previous discussion the current plan is still to
get HA into a minor release of 0.23 right?

Thanks,
Eli

Eli wrote:
> Each change was done individually with it's own jira, patch, and
> review.  People can review the sub-tasks if they don't want to look at
> the entire patch. The majority of the changes were reviewed before
> they were committed, when that wasn't the case review feedback was
> incorporated in follow-on patches. I don't see how this is any
> different from eg how federation was developed.
That's super.

The question at hand is does the community want to go ahead with the
1073 model, or not?  My preference is not, but since some in the
community like it, I'd suggest making a small tweak to bring it closer
in line with the intent of our bylaws, which clearly are RTC: "The
code can be committed after the first +1."  It's not productive to
re-litigate the individual commits on 1073.
-jg

On Mon, Jul 11, 2011 at 3:05 PM, Jakob Homan <[EMAIL PROTECTED]> wrote:
> Eli wrote:
>> Each change was done individually with it's own jira, patch, and
>> review.  People can review the sub-tasks if they don't want to look at
>> the entire patch. The majority of the changes were reviewed before
>> they were committed, when that wasn't the case review feedback was
>> incorporated in follow-on patches. I don't see how this is any
>> different from eg how federation was developed.
> That's super.
>
> The question at hand is does the community want to go ahead with the
> 1073 model, or not?  My preference is not, but since some in the
> community like it, I'd suggest making a small tweak to bring it closer
> in line with the intent of our bylaws, which clearly are RTC: "The
> code can be committed after the first +1."  It's not productive to
> re-litigate the individual commits on 1073.

This is the *branch* policy, which if I understand correctly, the
branch maintainer sets. For MR-279 Arun used CTR, for HDFS-1052 Suresh
used RTC. I think that's OK. IMO the people doing the work on the
branch should make the call. Either way the final patch to trunk goes
by the normal rules that any patch to trunk follows.

Thanks,
Eli

Deep sigh.  I'm not blocking RTC on the branch.  The branch maintainer
gets to decide that.  I'm not personally a big fan of it, but since
it's not my branch, it's not my call.
> Either way the final patch to trunk goes
> by the normal rules that any patch to trunk follows.

This is where I disagree.  RTC on individual commits increases the
chance for mistakes to slip in.  Branches that are the product of RTC
should have a stricter criteria for being applied to trunk,
particularly since they (nearly by definition) involve huge changes to
critical code paths.  That's why I've suggested (and all I've
suggested, contrary to what others have said) is that the final merge
get three +1s rather than the regular +1.  This is a good application
of "trust but verify" that will not slow down the development process
but will increase the chances that the class of bugs amenable to human
review will be lessened.
-jg

On Mon, Jul 11, 2011 at 4:05 PM, Jakob Homan <[EMAIL PROTECTED]> wrote:
> Deep sigh.  I'm not blocking RTC on the branch.  The branch maintainer
> gets to decide that.  I'm not personally a big fan of it, but since
> it's not my branch, it's not my call.
>> Either way the final patch to trunk goes
>> by the normal rules that any patch to trunk follows.
>
> This is where I disagree.

I'm fine requiring 3 +1s for the final merge, but as it stands today,
the normal rules apply.

Thanks,
Eli

On Mon, Jul 11, 2011 at 4:12 PM, Eli Collins <[EMAIL PROTECTED]> wrote:
> On Mon, Jul 11, 2011 at 4:05 PM, Jakob Homan <[EMAIL PROTECTED]> wrote:
>> Deep sigh.  I'm not blocking RTC on the branch.  The branch maintainer
>> gets to decide that.  I'm not personally a big fan of it, but since
>> it's not my branch, it's not my call.
>>> Either way the final patch to trunk goes
>>> by the normal rules that any patch to trunk follows.
>>
>> This is where I disagree.
>
> I'm fine requiring 3 +1s for the final merge, but as it stands today,
> the normal rules apply.

To be more clear, if you'd like to amend the bylaws so that a patch
that is a merge from a feature branch requires 3 +1s instead of the
normal single +1, go ahead, that's a separate discussion/vote that I
think people would support.

This discussion is just about the branching for a new HDFS-1623
branch, not about changing the rules for the project.

Thanks,
Eli

s/RTC/CTR/g. oy.

On Mon, Jul 11, 2011 at 4:05 PM, Jakob Homan <[EMAIL PROTECTED]> wrote:
> Deep sigh.  I'm not blocking RTC on the branch.  The branch maintainer
> gets to decide that.  I'm not personally a big fan of it, but since
> it's not my branch, it's not my call.
>> Either way the final patch to trunk goes
>> by the normal rules that any patch to trunk follows.
>
> This is where I disagree.  RTC on individual commits increases the
> chance for mistakes to slip in.  Branches that are the product of RTC
> should have a stricter criteria for being applied to trunk,
> particularly since they (nearly by definition) involve huge changes to
> critical code paths.  That's why I've suggested (and all I've
> suggested, contrary to what others have said) is that the final merge
> get three +1s rather than the regular +1.  This is a good application
> of "trust but verify" that will not slow down the development process
> but will increase the chances that the class of bugs amenable to human
> review will be lessened.
> -jg
>

On Jul 11, 2011, at 3:50 PM, Eli Collins wrote:

> On Mon, Jul 11, 2011 at 3:05 PM, Jakob Homan <[EMAIL PROTECTED]> wrote:
>> Eli wrote:
>>> ....
>
> This is the *branch* policy, which if I understand correctly, the
> branch maintainer sets.
>
> Thanks,
> Eli
I do not understand the new role "branch maintainer". If this is to make decisions about the branch, content and direction, I am not in favor of that. I think it should be a collaborative process for folks working on the HA branch and not an individuals call.

 HA is a collaborative project based around the design that we published. Aaron has volunteered to help merge trunk into this branch; others working on the project can also help in merging. When it is finally merged into trunk we will go through the normal rules (as proposed by Jakob, 3+1s etc., assuming it passes).

On Wed, Jul 13, 2011 at 5:40 PM, sanjay Radia <[EMAIL PROTECTED]> wrote:
>
> On Jul 11, 2011, at 3:50 PM, Eli Collins wrote:
>
>> On Mon, Jul 11, 2011 at 3:05 PM, Jakob Homan <[EMAIL PROTECTED]> wrote:
>>> Eli wrote:
>>>> ....
>>
>> This is the *branch* policy, which if I understand correctly, the
>> branch maintainer sets.
>>
>> Thanks,
>> Eli
>
>
> I do not understand the new role "branch maintainer". If this is to make decisions about the branch, content and direction, I am not in favor of that. I think it should be a collaborative process for folks working on the HA branch and not an individuals call.

Completely agree - that should have read "branch maintainers" plural -
didn't mean to suggest a new role. Any committer should be able to
checking, help merge, etc a feature branch. This worked well for
branch-0.20-append.

Btw, we started [1] but never finished the adoption of the release
manager (RM) role, we should do that.  I've seen "release master" used
in a couple of places, which I assume is synonymous.

1. http://www.mail-archive.com/[EMAIL PROTECTED]/msg01458.html

>  HA is a collaborative project based around the design that we published. Aaron has volunteered to help merge trunk into this branch; others working on the project can also
>  help in merging. When it is finally merged into trunk we will go through the normal rules (as proposed by Jakob, 3+1s etc., assuming it passes).

That's my understanding as well.

Thanks,
Eli

Hey everyone,

Since it seems that several people are opposed to the modified CTR portion
of my initial proposal, I'd like to retract that.

Given that the proposed modified CTR is really trying to address a problem
that may not in fact occur in practice (slow reviews on the branch) let's
just go with RTC as is done on trunk, and if it turns out that waiting for
reviews is slowing down progress on the branch, we'll revisit the discussion
at a later date.

I hope this allays the concerns raised.

Best,
Aaron

--
Aaron T. Myers
Software Engineer, Cloudera

Hey everyone,

Just a heads up - I just created the branch in SVN and called it, rather
uncreatively, "HDFS-1623".

--
Aaron T. Myers
Software Engineer, Cloudera

On Mon, Jul 18, 2011 at 3:24 PM, Aaron T. Myers <[EMAIL PROTECTED]> wrote:

> Hey everyone,
>
> Since it seems that several people are opposed to the modified CTR portion
> of my initial proposal, I'd like to retract that.
>
> Given that the proposed modified CTR is really trying to address a problem
> that may not in fact occur in practice (slow reviews on the branch) let's
> just go with RTC as is done on trunk, and if it turns out that waiting for
> reviews is slowing down progress on the branch, we'll revisit the discussion
> at a later date.
>
> I hope this allays the concerns raised.
>
> Best,
>  Aaron
>
> --
> Aaron T. Myers
> Software Engineer, Cloudera
>

+1. As you said, lets revisit this, if progress is slow due to reviews being
slow.

On Mon, Jul 18, 2011 at 3:24 PM, Aaron T. Myers <[EMAIL PROTECTED]> wrote:

> Hey everyone,
>
> Since it seems that several people are opposed to the modified CTR portion
> of my initial proposal, I'd like to retract that.
>
> Given that the proposed modified CTR is really trying to address a problem
> that may not in fact occur in practice (slow reviews on the branch) let's
> just go with RTC as is done on trunk, and if it turns out that waiting for
> reviews is slowing down progress on the branch, we'll revisit the
> discussion
> at a later date.
>
> I hope this allays the concerns raised.
>
> Best,
> Aaron
>
> --
> Aaron T. Myers
> Software Engineer, Cloudera
>

On Mon, Jul 11, 2011 at 1:28 PM, Suresh Srinivas <[EMAIL PROTECTED]>wrote:

> My preference is to review before commit. Reviewing incremental changes is
> much
> more effective than reviewing one mega patch during merge of a branch.
>

I certainly am not in favor of only reviewing a mega-patch either, and
that's not what I intended to propose. As Eli has already said, CTR doesn't
mean the individual patches don't get reviewed, just that they can be
committed to the branch and then any review feedback can be incorporated
later.

Note: I'm not married to the modified CTR proposal, but it is my preference.
It seems to have worked well previously on MR-279 and HDFS-1073, so seems
like a good idea here as well. I don't see why this branch is meaningfully
different from those.
> Diverging solutions:
> Second unrelated problem to branching is - in HDFS-1623, the design defines
> common elements to build HA solution. The idea was, it could enable
> different
> solutions depending on how folks want to go about it. I would like to make
> sure that we are all converging, instead of diverging. This means some of
> the
> jiras still need to be discussed, especially the ones that make major
> structural changes, adds interfaces etc. I am not sure 24 hr is a long
> enough
> notice.
>

Fair point. How about 48 hours?

--
Aaron T. Myers
Software Engineer, Cloudera

> -----Original Message-----
> From: sanjay Radia [mailto:[EMAIL PROTECTED]]
> Sent: Monday, July 11, 2011 11:30 AM

[Eric Payne] stuff deleted
> Further many of the changes are independent of each other and a single
> 1623 branch may not make sense.

[Eric Payne] I would be in favor of multiple branches, even in cases where things may be related but separable. This would make merging back to trunk much easier.

> sanjay

Thanks,
-Eric
>
>
> On Jul 8, 2011, at 12:22 PM, Suresh Srinivas wrote:
>
> > Thanks Aaron for sending the details from the meeting that Sanjay
> > had arranged. I am very happy that you have
> > agreed to collaborate on our HA design published in HDFS-1623.
> >
> > This is a critical feature for HDFS. This proposal is not exactly what I
> had
> > envisioned. Why don't we meet again to
> > discuss and come up with a proposal for branching and development-
> process?
> >
> > If there are other folks who want to contribute to development of HDFS-
> 1623,
> > please reach out to me and I will send
> > the meeting details.