Suresh and I are still reviewing this design and patch.
The 3077 code along with  the code pulled from 3092 is fairly substrantial. The design is also fairly complex and involved.
I  would request that we postpone the merge for another week to give folks time to review this fully.
sanjay

On Sep 25, 2012, at 4:02 PM, Todd Lipcon wrote:

> Dear fellow HDFS developers,
>
> Per my email thread last week ("Heads up: merge for QJM branch soon"
> at http://markmail.org/message/vkyh5culdsuxdb6t) I would like to
> propose merging the HDFS-3077 branch into trunk. The branch has been
> active since mid July and has stabilized significantly over the last
> two months. It has passed the full test suite, findbugs, and release
> audit, and I think it's ready to merge at this point.
>
> The branch has been fully developed using the standard
> 'review-then-commit' (RTC) policy, and the design is described in
> detail in a document attached to HDFS-3077 itself. The code itself has
> been contributed by me, Aaron, and Eli, but I'd be remiss not to also
> acknowledge the contributions to the design from discussions with
> Suresh, Sanjay, Henry Robinson, Patrick Hunt, Ivan Kelly, Andrew
> Purtell, Flavio Junqueira, Ben Reed, Nicholas, Bikas, Brandon, and
> others. Additionally, special thanks to Andrew Purtell and Stephen Chu
> for their help with cluster testing.
>
> This initial VOTE is to merge only into trunk, but, following the
> pattern of automatic failover, I expect to merge it into branch-2
> within a few weeks as well. The merge to branch-2 should be clean, as
> both I and Andrew Purtell have been testing on branch-2-derived
> codebases in addition to trunk.
>
> Please cast your vote by EOD Friday 9/29. Given that the branch has
> only had small changes in the last few weeks, and there was a "heads
> up" last week, I trust this should be enough time for committers to
> cast their votes. Per our by-laws, we need a minimum of three binding
> +1 votes from committers.
>
> I will start the voting with my own +1.
>
> Thanks
> -Todd
> --
> Todd Lipcon
> Software Engineer, Cloudera

Hey Sanjay,

While I understand it's substantial and complex code, the code and the
design doc have been available for several months, and the community
has certainly been aware of its development. I also gave a heads up
last week that I would call a merge this week. So I feel like there
has been sufficient time for interested parties to review.

That said, since I was sick for much of this week and not immediately
responsive to some of the questions from you and Suresh, I'm happy to
agree to postpone the merge to early next week. Let's extend the vote
to last until Monday end of day PST.

Of course if there are follow-up questions or bugs found after the
merge, you've all got my phone number and I'm not going anywhere! ;-)

Thanks
-Todd

On Fri, Sep 28, 2012 at 12:06 PM, sanjay Radia <[EMAIL PROTECTED]> wrote:
> Suresh and I are still reviewing this design and patch.
> The 3077 code along with  the code pulled from 3092 is fairly substrantial. The design is also fairly complex and involved.
> I  would request that we postpone the merge for another week to give folks time to review this fully.
>
>
> sanjay
>
>
>
>
>
> On Sep 25, 2012, at 4:02 PM, Todd Lipcon wrote:
>
>> Dear fellow HDFS developers,
>>
>> Per my email thread last week ("Heads up: merge for QJM branch soon"
>> at http://markmail.org/message/vkyh5culdsuxdb6t) I would like to
>> propose merging the HDFS-3077 branch into trunk. The branch has been
>> active since mid July and has stabilized significantly over the last
>> two months. It has passed the full test suite, findbugs, and release
>> audit, and I think it's ready to merge at this point.
>>
>> The branch has been fully developed using the standard
>> 'review-then-commit' (RTC) policy, and the design is described in
>> detail in a document attached to HDFS-3077 itself. The code itself has
>> been contributed by me, Aaron, and Eli, but I'd be remiss not to also
>> acknowledge the contributions to the design from discussions with
>> Suresh, Sanjay, Henry Robinson, Patrick Hunt, Ivan Kelly, Andrew
>> Purtell, Flavio Junqueira, Ben Reed, Nicholas, Bikas, Brandon, and
>> others. Additionally, special thanks to Andrew Purtell and Stephen Chu
>> for their help with cluster testing.
>>
>> This initial VOTE is to merge only into trunk, but, following the
>> pattern of automatic failover, I expect to merge it into branch-2
>> within a few weeks as well. The merge to branch-2 should be clean, as
>> both I and Andrew Purtell have been testing on branch-2-derived
>> codebases in addition to trunk.
>>
>> Please cast your vote by EOD Friday 9/29. Given that the branch has
>> only had small changes in the last few weeks, and there was a "heads
>> up" last week, I trust this should be enough time for committers to
>> cast their votes. Per our by-laws, we need a minimum of three binding
>> +1 votes from committers.
>>
>> I will start the voting with my own +1.
>>
>> Thanks
>> -Todd
>> --
>> Todd Lipcon
>> Software Engineer, Cloudera
>

--
Todd Lipcon
Software Engineer, Cloudera

Todd,
  Even though this work was under development over a period of time,  during its development it was not clear when the design was fairly stable to begin a thorough review. Hence the time of merge is when the real review happens in such large projects.

I have already indicated on the jira that i do not have any philosophical objection to this work being in HDFS - hence this should not be a worry on your part.

The extra week will result in a more through review (hopefully this will have a side effect of perhaps easing Konstanine's concern about
HDFS adding such complex code).

Lets plan to do the merge next monday.

thanks

sanjay
On Sep 28, 2012, at 3:02 PM, Todd Lipcon wrote:

> Hey Sanjay,
>
> While I understand it's substantial and complex code, the code and the
> design doc have been available for several months, and the community
> has certainly been aware of its development. I also gave a heads up
> last week that I would call a merge this week. So I feel like there
> has been sufficient time for interested parties to review.
>
> That said, since I was sick for much of this week and not immediately
> responsive to some of the questions from you and Suresh, I'm happy to
> agree to postpone the merge to early next week. Let's extend the vote
> to last until Monday end of day PST.
>
> Of course if there are follow-up questions or bugs found after the
> merge, you've all got my phone number and I'm not going anywhere! ;-)
>
> Thanks
> -Todd
>
> On Fri, Sep 28, 2012 at 12:06 PM, sanjay Radia <[EMAIL PROTECTED]> wrote:
>> Suresh and I are still reviewing this design and patch.
>> The 3077 code along with  the code pulled from 3092 is fairly substrantial. The design is also fairly complex and involved.
>> I  would request that we postpone the merge for another week to give folks time to review this fully.
>>
>>
>> sanjay
>>
>>
>>
>>
>>
>> On Sep 25, 2012, at 4:02 PM, Todd Lipcon wrote:
>>
>>> Dear fellow HDFS developers,
>>>
>>> Per my email thread last week ("Heads up: merge for QJM branch soon"
>>> at http://markmail.org/message/vkyh5culdsuxdb6t) I would like to
>>> propose merging the HDFS-3077 branch into trunk. The branch has been
>>> active since mid July and has stabilized significantly over the last
>>> two months. It has passed the full test suite, findbugs, and release
>>> audit, and I think it's ready to merge at this point.
>>>
>>> The branch has been fully developed using the standard
>>> 'review-then-commit' (RTC) policy, and the design is described in
>>> detail in a document attached to HDFS-3077 itself. The code itself has
>>> been contributed by me, Aaron, and Eli, but I'd be remiss not to also
>>> acknowledge the contributions to the design from discussions with
>>> Suresh, Sanjay, Henry Robinson, Patrick Hunt, Ivan Kelly, Andrew
>>> Purtell, Flavio Junqueira, Ben Reed, Nicholas, Bikas, Brandon, and
>>> others. Additionally, special thanks to Andrew Purtell and Stephen Chu
>>> for their help with cluster testing.
>>>
>>> This initial VOTE is to merge only into trunk, but, following the
>>> pattern of automatic failover, I expect to merge it into branch-2
>>> within a few weeks as well. The merge to branch-2 should be clean, as
>>> both I and Andrew Purtell have been testing on branch-2-derived
>>> codebases in addition to trunk.
>>>
>>> Please cast your vote by EOD Friday 9/29. Given that the branch has
>>> only had small changes in the last few weeks, and there was a "heads
>>> up" last week, I trust this should be enough time for committers to
>>> cast their votes. Per our by-laws, we need a minimum of three binding
>>> +1 votes from committers.
>>>
>>> I will start the voting with my own +1.
>>>
>>> Thanks
>>> -Todd
>>> --
>>> Todd Lipcon
>>> Software Engineer, Cloudera
>>
>
>
>
> --
> Todd Lipcon
> Software Engineer, Cloudera

Hi Sanjay,

The 7 extra days you requested beyond the original 7-day merge vote have
now elapsed, and we have the requisite three binding +1s to merge.

I'll plan to merge this late tonight unless there are any vetoes in the
meantime.

Of course we can continue to discuss the design and improve the clarity of
the documentation after it's in trunk, and if there's some kind of bug I'll
treat it as highest priority even after the merge.

Thanks
-Todd
On Mon, Oct 1, 2012 at 10:55 AM, sanjay Radia <[EMAIL PROTECTED]>wrote:

> Todd,
>   Even though this work was under development over a period of time,
>  during its development it was not clear when the design was fairly stable
> to begin a thorough review. Hence the time of merge is when the real review
> happens in such large projects.
>
> I have already indicated on the jira that i do not have any philosophical
> objection to this work being in HDFS - hence this should not be a worry on
> your part.
>
> The extra week will result in a more through review (hopefully this will
> have a side effect of perhaps easing Konstanine's concern about
> HDFS adding such complex code).
>
> Lets plan to do the merge next monday.
>
> thanks
>
> sanjay
>
>
>
>
> On Sep 28, 2012, at 3:02 PM, Todd Lipcon wrote:
>
> > Hey Sanjay,
> >
> > While I understand it's substantial and complex code, the code and the
> > design doc have been available for several months, and the community
> > has certainly been aware of its development. I also gave a heads up
> > last week that I would call a merge this week. So I feel like there
> > has been sufficient time for interested parties to review.
> >
> > That said, since I was sick for much of this week and not immediately
> > responsive to some of the questions from you and Suresh, I'm happy to
> > agree to postpone the merge to early next week. Let's extend the vote
> > to last until Monday end of day PST.
> >
> > Of course if there are follow-up questions or bugs found after the
> > merge, you've all got my phone number and I'm not going anywhere! ;-)
> >
> > Thanks
> > -Todd
> >
> > On Fri, Sep 28, 2012 at 12:06 PM, sanjay Radia <[EMAIL PROTECTED]>
> wrote:
> >> Suresh and I are still reviewing this design and patch.
> >> The 3077 code along with  the code pulled from 3092 is fairly
> substrantial. The design is also fairly complex and involved.
> >> I  would request that we postpone the merge for another week to give
> folks time to review this fully.
> >>
> >>
> >> sanjay
> >>
> >>
> >>
> >>
> >>
> >> On Sep 25, 2012, at 4:02 PM, Todd Lipcon wrote:
> >>
> >>> Dear fellow HDFS developers,
> >>>
> >>> Per my email thread last week ("Heads up: merge for QJM branch soon"
> >>> at http://markmail.org/message/vkyh5culdsuxdb6t) I would like to
> >>> propose merging the HDFS-3077 branch into trunk. The branch has been
> >>> active since mid July and has stabilized significantly over the last
> >>> two months. It has passed the full test suite, findbugs, and release
> >>> audit, and I think it's ready to merge at this point.
> >>>
> >>> The branch has been fully developed using the standard
> >>> 'review-then-commit' (RTC) policy, and the design is described in
> >>> detail in a document attached to HDFS-3077 itself. The code itself has
> >>> been contributed by me, Aaron, and Eli, but I'd be remiss not to also
> >>> acknowledge the contributions to the design from discussions with
> >>> Suresh, Sanjay, Henry Robinson, Patrick Hunt, Ivan Kelly, Andrew
> >>> Purtell, Flavio Junqueira, Ben Reed, Nicholas, Bikas, Brandon, and
> >>> others. Additionally, special thanks to Andrew Purtell and Stephen Chu
> >>> for their help with cluster testing.
> >>>
> >>> This initial VOTE is to merge only into trunk, but, following the
> >>> pattern of automatic failover, I expect to merge it into branch-2
> >>> within a few weeks as well. The merge to branch-2 should be clean, as
> >>> both I and Andrew Purtell have been testing on branch-2-derived
> >>> codebases in addition to trunk.
Todd Lipcon
Software Engineer, Cloudera

Todd,

As I indicated in my comments on the jira, I think some of the design
discussions and further simplification of design should happen before the
merge. See -
https://issues.apache.org/jira/browse/HDFS-3077?focusedCommentId=13470680&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13470680

I understand that you have spent a lot of time working on this. You have
indicated that you do not want to make any further design improvements. I
am willing to help by doing any improvements that comes out of the
discussions on the jira in 3077 branch and keep it up to date. I am also
willing to merge this branch into trunk. So my vote is to hold off the
merge until the discussions complete.

Regards,
Suresh

On Mon, Oct 8, 2012 at 5:46 PM, Todd Lipcon <[EMAIL PROTECTED]> wrote:

> Hi Sanjay,
>
> The 7 extra days you requested beyond the original 7-day merge vote have
> now elapsed, and we have the requisite three binding +1s to merge.
>
> I'll plan to merge this late tonight unless there are any vetoes in the
> meantime.
>
> Of course we can continue to discuss the design and improve the clarity of
> the documentation after it's in trunk, and if there's some kind of bug I'll
> treat it as highest priority even after the merge.
>
> Thanks
> -Todd
>
>
> On Mon, Oct 1, 2012 at 10:55 AM, sanjay Radia <[EMAIL PROTECTED]
> >wrote:
>
> > Todd,
> >   Even though this work was under development over a period of time,
> >  during its development it was not clear when the design was fairly
> stable
> > to begin a thorough review. Hence the time of merge is when the real
> review
> > happens in such large projects.
> >
> > I have already indicated on the jira that i do not have any philosophical
> > objection to this work being in HDFS - hence this should not be a worry
> on
> > your part.
> >
> > The extra week will result in a more through review (hopefully this will
> > have a side effect of perhaps easing Konstanine's concern about
> > HDFS adding such complex code).
> >
> > Lets plan to do the merge next monday.
> >
> > thanks
> >
> > sanjay
> >
> >
> >
> >
> > On Sep 28, 2012, at 3:02 PM, Todd Lipcon wrote:
> >
> > > Hey Sanjay,
> > >
> > > While I understand it's substantial and complex code, the code and the
> > > design doc have been available for several months, and the community
> > > has certainly been aware of its development. I also gave a heads up
> > > last week that I would call a merge this week. So I feel like there
> > > has been sufficient time for interested parties to review.
> > >
> > > That said, since I was sick for much of this week and not immediately
> > > responsive to some of the questions from you and Suresh, I'm happy to
> > > agree to postpone the merge to early next week. Let's extend the vote
> > > to last until Monday end of day PST.
> > >
> > > Of course if there are follow-up questions or bugs found after the
> > > merge, you've all got my phone number and I'm not going anywhere! ;-)
> > >
> > > Thanks
> > > -Todd
> > >
> > > On Fri, Sep 28, 2012 at 12:06 PM, sanjay Radia <[EMAIL PROTECTED]
> >
> > wrote:
> > >> Suresh and I are still reviewing this design and patch.
> > >> The 3077 code along with  the code pulled from 3092 is fairly
> > substrantial. The design is also fairly complex and involved.
> > >> I  would request that we postpone the merge for another week to give
> > folks time to review this fully.
> > >>
> > >>
> > >> sanjay
> > >>
> > >>
> > >>
> > >>
> > >>
> > >> On Sep 25, 2012, at 4:02 PM, Todd Lipcon wrote:
> > >>
> > >>> Dear fellow HDFS developers,
> > >>>
> > >>> Per my email thread last week ("Heads up: merge for QJM branch soon"
> > >>> at http://markmail.org/message/vkyh5culdsuxdb6t) I would like to
> > >>> propose merging the HDFS-3077 branch into trunk. The branch has been
> > >>> active since mid July and has stabilized significantly over the last
> > >>> two months. It has passed the full test suite, findbugs, and release
> > >>> audit, and I think it's ready to merge at this point.

http://hortonworks.com/download/

On Mon, Oct 8, 2012 at 6:01 PM, Suresh Srinivas <[EMAIL PROTECTED]>wrote:

> Todd,
>
> As I indicated in my comments on the jira, I think some of the design
> discussions and further simplification of design should happen before the
> merge. See -
>
> https://issues.apache.org/jira/browse/HDFS-3077?focusedCommentId=13470680&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13470680
>
>
I still haven't heard any actual concrete suggestion for a design
improvement. Just abstract notions that "it should be simpler". I could say
the same about several other features that have been done in the past (e.g
federation or YARN) but chose not to because I didn't participate in the
development. I generally have faith that, if several other people spent
several months working on a project, then there must be good reasons for
the complexity that I'm probably overlooking.

Several of us (not just I) spent lots of time on this over the last several
months, with all of the work going on in the open. My issue with this
discussion happening now is that no one saw fit to raise these questions
months ago when the design was first proposed. For example, this most
recent question about whether NewEpoch and PrepareRecovery should be
separate RPCs could have been raised in late June when the code in question
was first introduced as a patch.

Nor was anything raised when I gave a "heads up" that the branch was
complete and nearly ready for merge ~3 weeks ago. Only once I actually
called a merge vote has this discussion started. That doesn't seem like a
healthy way to do development.
> I understand that you have spent a lot of time working on this. You have
> indicated that you do not want to make any further design improvements. I
> am willing to help by doing any improvements that comes out of the
> discussions on the jira in 3077 branch and keep it up to date. I am also
> willing to merge this branch into trunk. So my vote is to hold off the
> merge until the discussions complete.
>

What are the criteria, then, for merging? I don't think it's possible to
definitively "prove" that a design is "simple". At some level it's a matter
of taste. So if the current design works, and is tested, and has people
signed up to support it and run it, why not merge? Just like any other part
of HDFS, we can continue to _improve_ it after it is in trunk. There are
many features that we commit and then improve later on when someone comes
up with a way to simplify it.

To be clear about the current status of the vote: you are officially
vetoing the merge?

Thanks
-Todd

 On Mon, Oct 8, 2012 at 5:46 PM, Todd Lipcon <[EMAIL PROTECTED]> wrote:
>
> > Hi Sanjay,
> >
> > The 7 extra days you requested beyond the original 7-day merge vote have
> > now elapsed, and we have the requisite three binding +1s to merge.
> >
> > I'll plan to merge this late tonight unless there are any vetoes in the
> > meantime.
> >
> > Of course we can continue to discuss the design and improve the clarity
> of
> > the documentation after it's in trunk, and if there's some kind of bug
> I'll
> > treat it as highest priority even after the merge.
> >
> > Thanks
> > -Todd
> >
> >
> > On Mon, Oct 1, 2012 at 10:55 AM, sanjay Radia <[EMAIL PROTECTED]
> > >wrote:
> >
> > > Todd,
> > >   Even though this work was under development over a period of time,
> > >  during its development it was not clear when the design was fairly
> > stable
> > > to begin a thorough review. Hence the time of merge is when the real
> > review
> > > happens in such large projects.
> > >
> > > I have already indicated on the jira that i do not have any
> philosophical
> > > objection to this work being in HDFS - hence this should not be a worry
> > on
> > > your part.
> > >
> > > The extra week will result in a more through review (hopefully this
> will
> > > have a side effect of perhaps easing Konstanine's concern about
> > > HDFS adding such complex code).
> > >
> > > Lets plan to do the merge next monday.

Todd Lipcon
Software Engineer, Cloudera

On Mon, Oct 8, 2012 at 6:20 PM, Todd Lipcon <[EMAIL PROTECTED]> wrote:

> On Mon, Oct 8, 2012 at 6:01 PM, Suresh Srinivas <[EMAIL PROTECTED]
> >wrote:
>
> > Todd,
> >
> > As I indicated in my comments on the jira, I think some of the design
> > discussions and further simplification of design should happen before the
> > merge. See -
> >
> >
> https://issues.apache.org/jira/browse/HDFS-3077?focusedCommentId=13470680&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13470680
> >
> >
> I still haven't heard any actual concrete suggestion for a design
> improvement. Just abstract notions that "it should be simpler". I could say
> the same about several other features that have been done in the past (e.g
> federation or YARN) but chose not to because I didn't participate in the
> development. I generally have faith that, if several other people spent
> several months working on a project, then there must be good reasons for
> the complexity that I'm probably overlooking.
>

I think I have participated enough in this work. Merge time seemed like a
good time
to review this more thoroughly given this jira has required quite a
bit of educating myself and spending a lot of my time on this because of
need to
understand the complexities/subtletie how paxos and ZAB applies to the
namenode
journals.

Please do not misunderstand that I am trying to block this work from being
merged. I am
supportive of this as you have seen in my previous response in this thread.
All I want
to see is if we can conclude the discussions and incorporate some the
comments that come
up after it.

>
> Several of us (not just I) spent lots of time on this over the last several
> months, with all of the work going on in the open. My issue with this
> discussion happening now is that no one saw fit to raise these questions
> months ago when the design was first proposed. For example, this most
> recent question about whether NewEpoch and PrepareRecovery should be
> separate RPCs could have been raised in late June when the code in question
> was first introduced as a patch.
>
> Nor was anything raised when I gave a "heads up" that the branch was
> complete and nearly ready for merge ~3 weeks ago. Only once I actually
> called a merge vote has this discussion started. That doesn't seem like a
> healthy way to do development.
>

Again Todd, you are reading more than what is intended. As I said earlier
merge
time is a good time to have complete picture and an opportunity to look at
final design
and implementation.
> What are the criteria, then, for merging? I don't think it's possible to
> definitively "prove" that a design is "simple". At some level it's a matter
> of taste. So if the current design works, and is tested, and has people
> signed up to support it and run it, why not merge? Just like any other part
> of HDFS, we can continue to _improve_ it after it is in trunk. There are
> many features that we commit and then improve later on when someone comes
> up with a way to simplify it.
>

One concern I have is, once this is merged into trunk I feel that any
proposed
improvements may be blocked. Given that why not just wait for these
discussions
to complete and do the work in this branch?

What is the need for getting this into trunk immediately?

If the problem is one of investment of your time, I have already offered to
help out.
> To be clear about the current status of the vote: you are officially
> vetoing the merge?
For now I am going to vote not to merge until the discussion completes.

Regards,
Suresh

On Mon, Oct 8, 2012 at 7:21 PM, Suresh Srinivas <[EMAIL PROTECTED]> wrote:
> On Mon, Oct 8, 2012 at 6:20 PM, Todd Lipcon <[EMAIL PROTECTED]> wrote:
>
>> On Mon, Oct 8, 2012 at 6:01 PM, Suresh Srinivas <[EMAIL PROTECTED]
>> >wrote:
>>
>> > Todd,
>> >
>> > As I indicated in my comments on the jira, I think some of the design
>> > discussions and further simplification of design should happen before the
>> > merge. See -
>> >
>> >
>> https://issues.apache.org/jira/browse/HDFS-3077?focusedCommentId=13470680&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13470680
>> >
>> >
>> I still haven't heard any actual concrete suggestion for a design
>> improvement. Just abstract notions that "it should be simpler". I could say
>> the same about several other features that have been done in the past (e.g
>> federation or YARN) but chose not to because I didn't participate in the
>> development. I generally have faith that, if several other people spent
>> several months working on a project, then there must be good reasons for
>> the complexity that I'm probably overlooking.
>>
>
> I think I have participated enough in this work. Merge time seemed like a
> good time
> to review this more thoroughly given this jira has required quite a
> bit of educating myself and spending a lot of my time on this because of
> need to
> understand the complexities/subtletie how paxos and ZAB applies to the
> namenode
> journals.
>
> Please do not misunderstand that I am trying to block this work from being
> merged. I am
> supportive of this as you have seen in my previous response in this thread.
> All I want
> to see is if we can conclude the discussions and incorporate some the
> comments that come
> up after it.
>
>>
>> Several of us (not just I) spent lots of time on this over the last several
>> months, with all of the work going on in the open. My issue with this
>> discussion happening now is that no one saw fit to raise these questions
>> months ago when the design was first proposed. For example, this most
>> recent question about whether NewEpoch and PrepareRecovery should be
>> separate RPCs could have been raised in late June when the code in question
>> was first introduced as a patch.
>>
>> Nor was anything raised when I gave a "heads up" that the branch was
>> complete and nearly ready for merge ~3 weeks ago. Only once I actually
>> called a merge vote has this discussion started. That doesn't seem like a
>> healthy way to do development.
>>
>
> Again Todd, you are reading more than what is intended. As I said earlier
> merge
> time is a good time to have complete picture and an opportunity to look at
> final design
> and implementation.
>
>
>> What are the criteria, then, for merging? I don't think it's possible to
>> definitively "prove" that a design is "simple". At some level it's a matter
>> of taste. So if the current design works, and is tested, and has people
>> signed up to support it and run it, why not merge? Just like any other part
>> of HDFS, we can continue to _improve_ it after it is in trunk. There are
>> many features that we commit and then improve later on when someone comes
>> up with a way to simplify it.
>>
>
> One concern I have is, once this is merged into trunk I feel that any
> proposed
> improvements may be blocked.

That could be used as justification for not merging any significant
feature.  We've improved HA and federation after they were merged,
there's no reason we won't do the same for QJM.

> Given that why not just wait for these
> discussions
> to complete and do the work in this branch?

We have already delayed it a week per your request, which you agreed
to, and we've already had a vote that passed with three bindings +1s.
That's the protocol of the project - which you should respect. We're
not stopping the merge just because you're afraid some of your
improvements might not be incorporated.

Thanks,
Eli

I have posted a comment on 3077 listing  two areas that i would like to fix in the QJM.
If we agree on that we can merge now and fix those two later.
Todd has volunteered to fix one and I or suresh can fix the other.
sanjay

On Oct 9, 2012, at 1:44 PM, sanjay Radia wrote:

> I have posted a comment on 3077 listing  two areas that i would like to fix in the QJM.
> If we agree on that we can merge now and fix those two later.
> Todd has volunteered to fix one and I or suresh can fix the other.
>
>
> sanjay
>

2 jiras have been created  (see 3077 for the details).
+1 for the merge. The 2 liras will be completed after the merge.
Sorry for taking an extra 2 days for my +1.
Todd, thanks for the work you have put into this jira.
(This Jira has had the side effect of forcing me to come up to speed on Paxos and Zab. :-) )

sanjay

Given the completion of discussions and the resolution towards next steps I
am +1 for the merge.

On Wed, Oct 10, 2012 at 7:17 PM, sanjay Radia <[EMAIL PROTECTED]>wrote:

>
> On Oct 9, 2012, at 1:44 PM, sanjay Radia wrote:
>
> > I have posted a comment on 3077 listing  two areas that i would like to
> fix in the QJM.
> > If we agree on that we can merge now and fix those two later.
> > Todd has volunteered to fix one and I or suresh can fix the other.
> >
> >
> > sanjay
> >
>
> 2 jiras have been created  (see 3077 for the details).
> +1 for the merge. The 2 liras will be completed after the merge.
> Sorry for taking an extra 2 days for my +1.
> Todd, thanks for the work you have put into this jira.
> (This Jira has had the side effect of forcing me to come up to speed on
> Paxos and Zab. :-) )
>
> sanjay
--
http://hortonworks.com/download/

Thanks, Sanjay and Suresh.

I've merged the branch into trunk.
Will take care of merging the CHANGES.txt lists, etc, tomorrow, and start
working on the agreed-upon improvements early next week.

Thanks
-Todd

On Wed, Oct 10, 2012 at 9:10 PM, Suresh Srinivas <[EMAIL PROTECTED]>wrote:

> Given the completion of discussions and the resolution towards next steps I
> am +1 for the merge.
>
> On Wed, Oct 10, 2012 at 7:17 PM, sanjay Radia <[EMAIL PROTECTED]
> >wrote:
>
> >
> > On Oct 9, 2012, at 1:44 PM, sanjay Radia wrote:
> >
> > > I have posted a comment on 3077 listing  two areas that i would like to
> > fix in the QJM.
> > > If we agree on that we can merge now and fix those two later.
> > > Todd has volunteered to fix one and I or suresh can fix the other.
> > >
> > >
> > > sanjay
> > >
> >
> > 2 jiras have been created  (see 3077 for the details).
> > +1 for the merge. The 2 liras will be completed after the merge.
> > Sorry for taking an extra 2 days for my +1.
> > Todd, thanks for the work you have put into this jira.
> > (This Jira has had the side effect of forcing me to come up to speed on
> > Paxos and Zab. :-) )
> >
> > sanjay
>
>
>
>
> --
> http://hortonworks.com/download/
>

--
Todd Lipcon
Software Engineer, Cloudera

Our position on the QJM is we've already "taken delivery" from the feature
branch and will maintain a private HDFS fork of branch-2 if necessary, i.e.
we don't have a significant stake in this discussion except at a meta level
as potential contributors.

This is a case study of why feature branch development is problematic? A
would-be contributor can make a significant effort over months and end up
with a baked result, ready to move on to a perhaps large backlog of other
work. Others can reasonably be expected meanwhile to triage their attention
until the merge point. Then, reconsidering design points will be more
challenging than a design discussion at an earlier time, because there is
already a significant sunk cost in the current design. What does a feature
branch buy here over development in situ in trunk (like the BookKeeper JM)?
Should would-be future contributors consider a feature branch as a viable
route toward contribution or not?

On Tuesday, October 9, 2012, Suresh Srinivas wrote:

> On Mon, Oct 8, 2012 at 6:20 PM, Todd Lipcon <[EMAIL PROTECTED]<javascript:;>>
> wrote:
>
> > On Mon, Oct 8, 2012 at 6:01 PM, Suresh Srinivas <[EMAIL PROTECTED]<javascript:;>
> > >wrote:
> >
> > > Todd,
> > >
> > > As I indicated in my comments on the jira, I think some of the design
> > > discussions and further simplification of design should happen before
> the
> > > merge. See -
> > >
> > >
> >
> https://issues.apache.org/jira/browse/HDFS-3077?focusedCommentId=13470680&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13470680
> > >
> > >
> > I still haven't heard any actual concrete suggestion for a design
> > improvement. Just abstract notions that "it should be simpler". I could
> say
> > the same about several other features that have been done in the past
> (e.g
> > federation or YARN) but chose not to because I didn't participate in the
> > development. I generally have faith that, if several other people spent
> > several months working on a project, then there must be good reasons for
> > the complexity that I'm probably overlooking.
> >
>
> I think I have participated enough in this work. Merge time seemed like a
> good time
> to review this more thoroughly given this jira has required quite a
> bit of educating myself and spending a lot of my time on this because of
> need to
> understand the complexities/subtletie how paxos and ZAB applies to the
> namenode
> journals.
>
> Please do not misunderstand that I am trying to block this work from being
> merged. I am
> supportive of this as you have seen in my previous response in this thread.
> All I want
> to see is if we can conclude the discussions and incorporate some the
> comments that come
> up after it.
>
> >
> > Several of us (not just I) spent lots of time on this over the last
> several
> > months, with all of the work going on in the open. My issue with this
> > discussion happening now is that no one saw fit to raise these questions
> > months ago when the design was first proposed. For example, this most
> > recent question about whether NewEpoch and PrepareRecovery should be
> > separate RPCs could have been raised in late June when the code in
> question
> > was first introduced as a patch.
> >
> > Nor was anything raised when I gave a "heads up" that the branch was
> > complete and nearly ready for merge ~3 weeks ago. Only once I actually
> > called a merge vote has this discussion started. That doesn't seem like a
> > healthy way to do development.
> >
>
> Again Todd, you are reading more than what is intended. As I said earlier
> merge
> time is a good time to have complete picture and an opportunity to look at
> final design
> and implementation.
>
>
> > What are the criteria, then, for merging? I don't think it's possible to
> > definitively "prove" that a design is "simple". At some level it's a
> matter
> > of taste. So if the current design works, and is tested, and has people
> > signed up to support it and run it, why not merge? Just like any other
Best regards,

   - Andy

Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)

On Mon, Oct 8, 2012 at 8:03 PM, Andrew Purtell <[EMAIL PROTECTED]> wrote:

> Our position on the QJM is we've already "taken delivery" from the feature
> branch and will maintain a private HDFS fork of branch-2 if necessary, i.e.
> we don't have a significant stake in this discussion except at a meta level
> as potential contributors.
Andrew, as I had said in person to you, thank you for helping in testing
this
feature. This is very useful for the community.
>
> This is a case study of why feature branch development is problematic? A
> would-be contributor can make a significant effort over months and end up
> with a baked result, ready to move on to a perhaps large backlog of other
> work. Others can reasonably be expected meanwhile to triage their attention
> until the merge point. Then, reconsidering design points will be more
> challenging than a design discussion at an earlier time, because there is
> already a significant sunk cost in the current design. What does a feature
> branch buy here over development in situ in trunk (like the BookKeeper JM)?
> Should would-be future contributors consider a feature branch as a viable
> route toward contribution or not?
Andrew, feature branches have worked quite well in HDFS. See many of the
feature developments that have happened in different branches. Let me
reiterate
again. I am not trying to block this work.

I have explained the reasons why merge time made sense for this jira, given
the complexity of design,
where it started with ZAB, then added paxos in the middle of the
development.
These protocols are quite complicated, with multiple papers published on
describing
protocol, the proof, how to implement it and various simplifications and
variations of the protocol.
Requiring intimate knowledge of this and applying it to namenode journals
is non
trivial. I and Sanjay have spent many hours every day last three weeks on
understanding this.
Not all features have this complicated code and protocols and hence do not
require this
level of review.

All I am suggesting here is that we wait for the design discussion to
complete. Based on my comments or Sanjay's comment, you would see that
no one is asking for complete redesign. Some of the subtle issues and their
solutions may not be clear to all. See the discussions in the jira and
compare it with
what is in the design. Second, based on the protocols used, the
questions that are being asked are, can step x and step y be merged. Other
questions
are around perceived diversions from the well known protocols and its
implementation
(example ZooKeeper).

At the end of these discussion there may be few or no changes required.

Regards,
Suresh

Sorry, just to be clear "our" and "we" below refer to my employer, nothing
to do with HBase. Please pardon any confusion.

On Tuesday, October 9, 2012, Andrew Purtell wrote:

> Our position on the QJM is we've already "taken delivery" from the feature
> branch and will maintain a private HDFS fork of branch-2 if necessary, i.e.
> we don't have a significant stake in this discussion except at a meta level
> as potential contributors.
>
> This is a case study of why feature branch development is problematic? A
> would-be contributor can make a significant effort over months and end up
> with a baked result, ready to move on to a perhaps large backlog of other
> work. Others can reasonably be expected meanwhile to triage their attention
> until the merge point. Then, reconsidering design points will be more
> challenging than a design discussion at an earlier time, because there is
> already a significant sunk cost in the current design. What does a feature
> branch buy here over development in situ in trunk (like the BookKeeper JM)?
> Should would-be future contributors consider a feature branch as a viable
> route toward contribution or not?
>
> On Tuesday, October 9, 2012, Suresh Srinivas wrote:
>
>> On Mon, Oct 8, 2012 at 6:20 PM, Todd Lipcon <[EMAIL PROTECTED]> wrote:
>>
>> > On Mon, Oct 8, 2012 at 6:01 PM, Suresh Srinivas <[EMAIL PROTECTED]
>> > >wrote:
>> >
>> > > Todd,
>> > >
>> > > As I indicated in my comments on the jira, I think some of the design
>> > > discussions and further simplification of design should happen before
>> the
>> > > merge. See -
>> > >
>> > >
>> >
>> https://issues.apache.org/jira/browse/HDFS-3077?focusedCommentId=13470680&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13470680
>> > >
>> > >
>> > I still haven't heard any actual concrete suggestion for a design
>> > improvement. Just abstract notions that "it should be simpler". I could
>> say
>> > the same about several other features that have been done in the past
>> (e.g
>> > federation or YARN) but chose not to because I didn't participate in the
>> > development. I generally have faith that, if several other people spent
>> > several months working on a project, then there must be good reasons for
>> > the complexity that I'm probably overlooking.
>> >
>>
>> I think I have participated enough in this work. Merge time seemed like a
>> good time
>> to review this more thoroughly given this jira has required quite a
>> bit of educating myself and spending a lot of my time on this because of
>> need to
>> understand the complexities/subtletie how paxos and ZAB applies to the
>> namenode
>> journals.
>>
>> Please do not misunderstand that I am trying to block this work from being
>> merged. I am
>> supportive of this as you have seen in my previous response in this
>> thread.
>> All I want
>> to see is if we can conclude the discussions and incorporate some the
>> comments that come
>> up after it.
>>
>> >
>> > Several of us (not just I) spent lots of time on this over the last
>> several
>> > months, with all of the work going on in the open. My issue with this
>> > discussion happening now is that no one saw fit to raise these questions
>> > months ago when the design was first proposed. For example, this most
>> > recent question about whether NewEpoch and PrepareRecovery should be
>> > separate RPCs could have been raised in late June when the code in
>> question
>> > was first introduced as a patch.
>> >
>> > Nor was anything raised when I gave a "heads up" that the branch was
>> > complete and nearly ready for merge ~3 weeks ago. Only once I actually
>> > called a merge vote has this discussion started. That doesn't seem like
>> a
>> > healthy way to do development.
>> >
>>
>> Again Todd, you are reading more than what is intended. As I said earlier
>> merge
>> time is a good time to have complete picture and an opportunity to look at
>> final design
>> and implementation.
>>
>>
>> > What are the criteria, then, for merging? I don't think it's possible to

Best regards,

   - Andy

Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)

Sanjay, Suresh,

I don't think we should block the merge on discussion about parts of
the design. It's great that you guys are coming up to speed on the
design and have feedback. We're confident the feedback can be
addressed in follow-on jiras (just as federation and HA have been
iterated on after merging).  Todd has maintained the 3077 branch for
months, continuously updated the design doc, addressed questions
promptly, given sufficient heads up, and even even delayed the merge
per your request.  If you have specific technical objections with the
merge patch you should raise them, otherwise we're going to merge it
and iterate on the feedback in the jira with follow on patches.

Thanks,
Eli
On Mon, Oct 1, 2012 at 10:55 AM, sanjay Radia <[EMAIL PROTECTED]> wrote:
> Todd,
>   Even though this work was under development over a period of time,  during its development it was not clear when the design was fairly stable to begin a thorough review. Hence the time of merge is when the real review happens in such large projects.
>
> I have already indicated on the jira that i do not have any philosophical objection to this work being in HDFS - hence this should not be a worry on your part.
>
> The extra week will result in a more through review (hopefully this will have a side effect of perhaps easing Konstanine's concern about
> HDFS adding such complex code).
>
> Lets plan to do the merge next monday.
>
> thanks
>
> sanjay
>
>
>
>
> On Sep 28, 2012, at 3:02 PM, Todd Lipcon wrote:
>
>> Hey Sanjay,
>>
>> While I understand it's substantial and complex code, the code and the
>> design doc have been available for several months, and the community
>> has certainly been aware of its development. I also gave a heads up
>> last week that I would call a merge this week. So I feel like there
>> has been sufficient time for interested parties to review.
>>
>> That said, since I was sick for much of this week and not immediately
>> responsive to some of the questions from you and Suresh, I'm happy to
>> agree to postpone the merge to early next week. Let's extend the vote
>> to last until Monday end of day PST.
>>
>> Of course if there are follow-up questions or bugs found after the
>> merge, you've all got my phone number and I'm not going anywhere! ;-)
>>
>> Thanks
>> -Todd
>>
>> On Fri, Sep 28, 2012 at 12:06 PM, sanjay Radia <[EMAIL PROTECTED]> wrote:
>>> Suresh and I are still reviewing this design and patch.
>>> The 3077 code along with  the code pulled from 3092 is fairly substrantial. The design is also fairly complex and involved.
>>> I  would request that we postpone the merge for another week to give folks time to review this fully.
>>>
>>>
>>> sanjay
>>>
>>>
>>>
>>>
>>>
>>> On Sep 25, 2012, at 4:02 PM, Todd Lipcon wrote:
>>>
>>>> Dear fellow HDFS developers,
>>>>
>>>> Per my email thread last week ("Heads up: merge for QJM branch soon"
>>>> at http://markmail.org/message/vkyh5culdsuxdb6t) I would like to
>>>> propose merging the HDFS-3077 branch into trunk. The branch has been
>>>> active since mid July and has stabilized significantly over the last
>>>> two months. It has passed the full test suite, findbugs, and release
>>>> audit, and I think it's ready to merge at this point.
>>>>
>>>> The branch has been fully developed using the standard
>>>> 'review-then-commit' (RTC) policy, and the design is described in
>>>> detail in a document attached to HDFS-3077 itself. The code itself has
>>>> been contributed by me, Aaron, and Eli, but I'd be remiss not to also
>>>> acknowledge the contributions to the design from discussions with
>>>> Suresh, Sanjay, Henry Robinson, Patrick Hunt, Ivan Kelly, Andrew
>>>> Purtell, Flavio Junqueira, Ben Reed, Nicholas, Bikas, Brandon, and
>>>> others. Additionally, special thanks to Andrew Purtell and Stephen Chu
>>>> for their help with cluster testing.
>>>>
>>>> This initial VOTE is to merge only into trunk, but, following the
>>>> pattern of automatic failover, I expect to merge it into branch-2

 I am in favor of 3077  being part of HDFS.
   - It is one of several  way of configuring HDFS HA (shared storage is an option or if we extend the BackupNode a little, then it will be  another option for HA.)
  -  I believe I can use part of 3092/3077  to move journals to the Datanodes down the road.
- I have a hard time seeing 3077 having independent value outside of HDFS unless a fair amount of work is added to 3077.
- I do agree with Konstantin that this is fairly complex work;  by merging it in we are adding a fair amount  of future maintenance work. I am reviewing the design and code to see if we can simplify parts of this.
On a separate note - response to some of the comments in this thread.
- Even though I am in favor of this becoming part of HDFS, I don't buy the argument that HDFS HA should not depend on external projects.  
    - HDFS HA needs ZK which is an external project. I also think it fine for folks to use Linux HA  - there to be a blueprint for HDFS HA with Linux HA for those who choose to use it.

I  never bought this argument in the case of BK either.
Wrt to BK I feel the HDFS team had done a  NIH. I believe we could have worked with the BK team to add/modify it as needed. The BK team was willing.
We originally started with a simpler design for the journal daemon/node and kept on finding bugs in the design and then ended up reinventing some of the BK work.
Yes i have read todd's arguments and have debated this with suresh at great length -  reasonable folks disagree at times )

To summarize - i am in favor of this being part of HDFS and am reviewing the design and code.
sanjay
On Sep 26, 2012, at 2:32 PM, Stack wrote:

> On Tue, Sep 25, 2012 at 11:21 PM, Konstantin Shvachko
> <[EMAIL PROTECTED]> wrote:
>> I think this is a great work, Todd.
>> And I think we should not merge it into trunk or other branches.
>> As I suggested earlier on this list I think this should be spinned off
>> as a separate project or a subproject.
>>
>
> I'd be -1 on that.
>
> Users shouldn't have to go elsewhere to get a fix for SPOF.
>
> St.Ack

On Thu, Sep 27, 2012 at 2:06 AM, Konstantin Shvachko
<[EMAIL PROTECTED]> wrote:
> The SPOF is in HDFS. This project is about shared storage
> implementation, that could be replaced by NFS or BookKeeper or
> something else.

You cannot equate QJM to a solution that requires an NFS filer.  A
filer is just not possible in the deploys I am privy to.

> Same if QJ failed. You go to the creators, which will be somewhat
> closer than NetApp, because it is still Hadoop.
>

You seem to be undoing yourself with the above setup Konstantin.  At
our deploy, we can't do NetApp so calling them will never happen.  If
a problem in QJM, it's Apache HDFS, so it'll be fixed by the community
-- hopefully w/ input by creators -- as any other issue would be fixed
(or not).

St.Ack

Sorry for the slowness in response - I've been in bed with a fever the
past couple days and only sporadically on email. I'll respond to a few
different points made in the above thread here:

Konst> We keep growing the HDFS umbrella with competing technologies
Konst> (http/web HDFS as an example) within it.

I suppose these are "competing technologies" in the sense that there
are multiple options to solve the same goal: NFS shared storage,
BookKeeper-based storage, and QJM-based storage. On the other hand,
they are quite distinct in my view:
- NFS shared storage: this simply re-uses existing code
(FileJournalManager) that we already have for the non-HA case. It has
several downsides for use in an HA cluster (see the HDFS-3077 design
doc for discussion of them).
- BK storage: this is advantageous for some people who already run
BookKeeper in their datacenter. However, I think for those not already
running it, it adds much complexity, and lacks several advantages of
the QJM - again, see the design discussion on HDFS-3077 for details.
The tldr version is: Hadoop-consistent metrics, security, IPC, and
configuration.
- QJM storage: I won't expand on the advantages here, please refer to
the design doc.

So, if they fulfill the same goal, why let them coexist in the codebase?

I would argue that this is one of those things that often happens in a
community-developed project: different users or vendors may have
slightly different requirements, and therefore prefer different
solutions. Some people already have an HA NFS filer and run all their
HA using NFS, so the NFS shared storage seems "safest" to them. Others
already run BookKeeper, so they feel most comfortable with that
approach (or they want to use BookKeeper for other replicated logs).
For me and a lot of our customers, QJM seems to be the best approach.

Rather than duking it out and trying to declare one the "winner", our
answer has generally been "make it pluggable, and people can pick the
implementation that's right for them". If at some point later, it
turns out that everyone's using the same plug-in, by all means we
should consider ejecting the others, eg to github. But while there are
several active committers who have pledged that they will maintain a
solution, it seems reasonable to keep it as part of HDFS core. In this
case, I'm certainly planning on maintaining it, and it seems like
others on the list are on board as well.

Konst> Which makes the project harder to stabilize and release.
Konst> Not touching MR/Yarn here.
Konst> My concern is that if we add another 6000 lines of code to Hadoop-2,
Konst> it will take yet another x months for stabilization.

As you pointed out in your first email, the new code is mostly
stand-alone. The changes to the core NN are really quite small, and
actually benefit all of the storage implementations (eg enhancing the
web UI to display non-file JMs). So, if you don't choose to add a
qjournal:// URI on your cluster, it won't affect your stability at
all.

On the other hand, I happen to believe it is as stable as the rest of
Hadoop. Several committers, as well as some folks in the community,
and our internal QA team have been testing it for a couple months now,
with extensive fault injection, real workloads, etc, and it has held
up nicely. Stability is of course subjective in some sense, so if you
want to think of it as unstable owing to the fact that it's new code,
I'll respect that. But I don't think you can reject new code from a
project just because it's new code.

Konst> While it is not clear why people cannot just use NFS filers for
shared storage,
Konst> as you originally designed.

This was always seen as a step along the way to the final solution. As
several people have said, NFS filers are not available in every
organization. It was a useful point along the way, as it provided a
working production-ready HA solution several months back, whereas
waiting for a non-NFS solution would have delayed it quite a bit. For
reference, check out Sanjay and Aaron's presentation from Hadoop World
last year: http://www.slideshare.net/cloudera/hadoop-world-2011-hdfs-name-node-high-availablity-aaron-myers-cloudera-sanjay-radia-hortonworks
 ("Other options to share NN metadata" under "Future Work")

Konst> Don't understand your argument. Else where?
Konst> One way or another users will be talking to Todd.

Not sure what you meant by this. Sure, I wrote a lot of the code, but
it's a contribution to the community, and in no way do I have a
monopoly on helping people use it (I certainly hope I don't!) The
branch to be merged includes full docs on how to get running, and I
certainly hope that other Hadoop vendors will ship and recommend this
option too. And those who choose to consume Hadoop directly from
Apache, rather than through a vendor, should be able to use it equally
well, without patching together bits and pieces from a number of
repositories.
Thanks
-Todd

On Thu, Sep 27, 2012 at 9:59 AM, Stack <[EMAIL PROTECTED]> wrote:

Todd Lipcon
Software Engineer, Cloudera

I am in favor of keeping QJM in HDFS.

QJM is very specific to HDFS and is tightly coupled with HDFS code,
essentially extending the current editlog functionality that writes to
local disk to writing to a separate set of daemons. Clearly there is a need
for this in HDFS. Konstantin, I see your point that it brings in
complexity. To certain extent this complexity cannot be avoided given the
goals and the feature set. Additionally people can chose not to use this
optional functionality and avoid the complexity it brings both in terms
more involved journaling mechanism and management of additional set of
daemons it introduces.

In future, we could chose to make this generic enough or standardise the
HDFS interfaces/code that it depends on and perhaps spin it off as another
project. Also if this is complex and error prone, perhaps we could simplify
the functionality or even replace it. But for now I feel it belongs to HDFS.

On Wed, Sep 26, 2012 at 10:50 AM, Todd Lipcon <[EMAIL PROTECTED]> wrote:

> On Tue, Sep 25, 2012 at 11:21 PM, Konstantin Shvachko
> <[EMAIL PROTECTED]> wrote:
> > I think this is a great work, Todd.
> > And I think we should not merge it into trunk or other branches.
> > As I suggested earlier on this list I think this should be spinned off
> > as a separate project or a subproject.
> >
> > - The code is well detached as a self contained package.
>
> The addition is mostly self-contained, but it makes use of a bunch of
> "private" parts of HDFS and Common:
> - Reuses all of the Hadoop security infrastructure, IPC, metrics, etc
> - Coupled to the JournalManager interface which is still evolving. In
> fact there were several patches in trunk which were done during the
> development of this project, specifically to make this API more
> general. There's still some further work to be done in this area on
> the generic interface -- eg support for upgrade/rollback.
> - The functional tests make use of a bunch of "private" HDFS APIs as well.
>
> > - It is a logically stand-alone project that can be replaced by other
> > technologies.
> > - If it is a separate project then there is no need to port it to
> > other versions. You can package it as a dependent jar.
>
> Per above, it's not that separate, because in order to build it, we
> had to make a number of changes to core HDFS internal interfaces. It
> currently couldn't be used to store anything except for NN logs. It
> would be a nice extension to truly separate it out into a
> content-agnostic quorum-based edit log, but today it actually uses the
> existing edit log validation code to determine valid lengths, etc.
>
> > - Finally, it will be a good precedent of spinning new projects out of
> > HDFS rather than bringing everything under HDFS umbrella.
> >
> > Todd, I had a feeling you were in favor of this direction?
>
> I'm not in favor of it - I had said previously that it's worth
> discussing if several other people believe the same.
>
> I know that we plan to ship it as part of CDH and will be our
> recommended way of running HA HDFS. If the community doesn't accept
> the contribution, and prefers that we maintain it in a fork on github,
> then it's worth hearing. But I imagine that many other community
> members will want to either use or it ship it as part of their
> distros. Moving it to an entirely separate standalone project will
> just add extra work for these folks who, like us, think it's currently
> the best option for HA log storage.
>
> If at some point in the future, the internal APIs have fully
> stabilized (security, IPC, edit log streams, JournalManager, metrics,
> etc) then we can pull it out at that time.
>
> -Todd
>
> > On Tue, Sep 25, 2012 at 4:58 PM, Eli Collins <[EMAIL PROTECTED]> wrote:
> >> +1   Awesome work Todd.
> >>
> >> On Tue, Sep 25, 2012 at 4:02 PM, Todd Lipcon <[EMAIL PROTECTED]> wrote:
> >>> Dear fellow HDFS developers,
> >>>
> >>> Per my email thread last week ("Heads up: merge for QJM branch soon"
> >>> at http://markmail.org/message/vkyh5culdsuxdb6t) I would like to

http://hortonworks.com/download/

It's true we can package and use HDFS and QJM, if it were separately
distributed, because we supply our internal users with an Apache Hadoop
distribution with our additional polish. Also, Cloudera or Hortonworks or
others could package them together, but then what of the Apache Hadoop
distribution itself. Apache Hadoop would be less user friendly than it has
to be. And the larger argument here, I think, is finally rectifying that
SPOF without introducing another? But that's just one users opinion...

On Thursday, September 27, 2012, Konstantin Shvachko wrote:

> On Thu, Sep 27, 2012 at 1:50 AM, Andrew Purtell <[EMAIL PROTECTED]<javascript:;>>
> wrote:
> > Speaking as an Apache Hadoop user who must do something with the NameNode
> > single point of failure this year, I don't subscribe to the view that
> > moving that SPOF from the NameNode to a NFS filer is reasonable to ask of
> > those not already set up with NetApp or similar, or those running in a
> > "cloud" environment, or those quite common deployments (sadly) where our
> > legacy datacenter designs are not... ideal. I would be curious how common
> > this opinion is (or not).
>
> Well I was arguing that same thing about NFS from the beginning of
> this HA design.
> I am glad to hear it now, even though its too late.
>
> > So we intend to run HDFS 2 in HA configuration using the QJM for edit log
> > persistence, fencing, and recovery.
>
> Whether it is in the core HDFS or not, right? Cos its a matter of
> packaging.
>
> > Also, there is a BookKeeper based journal manager under development
> already
> > in HDFS in trunk and on branch-2. Occasionally I've broken it patching up
> > HDFS. I suppose that should come out too? But I would think that not a
> good
> > idea either per the above reasoning.
>
> Competing technologies should exist outside the project.
> Let different distributions compete outside the core.
>
> --Konst
>
> > On Thursday, September 27, 2012, Konstantin Shvachko wrote:
> >
> >> Hi Todd,
> >>
> >> > I had said previously that it's worth
> >> > discussing if several other people believe the same.
> >>
> >> Well let's put it on to general list for discussion then?
> >> Seems to me an important issue for  Hadoop evolution in general.
> >> We keep growing the HDFS umbrella with competing technologies
> >> (http/web HDFS as an example) within it.
> >> Which makes the project harder to stabilize and release.
> >> Not touching MR/Yarn here.
> >>
> >> > If at some point in the future, the internal APIs have fully
> >> > stabilized (security, IPC, edit log streams, JournalManager, metrics,
> >> > etc) then we can pull it out at that time.
> >>
> >> By that time it will monolithically grow into HDFS and vise versa.
> >>
> >> > I know that we plan to ship it as part of CDH and will be our
> >> > recommended way of running HA HDFS.
> >>
> >> Sounds like CDH is moving well in release plans and otherwise.
> >> My concern is that if we add another 6000 lines of code to Hadoop-2,
> >> it will take yet another x months for stabilization.
> >> While it is not clear why people cannot just use NFS filers for shared
> >> storage,
> >> as you originally designed.
> >>
> >> > distros. Moving it to an entirely separate standalone project will
> >> > just add extra work for these folks who, like us, think it's currently
> >> > the best option for HA log storage.
> >>
> >> Don't know who these folks are. I see it as less work for HDFS
> community,
> >> because there is no need for porting and supporting this project in two
> or
> >> more different versions.
> >>
> >> Thanks,
> >> --Konstantin
> >>
> >> On Wed, Sep 26, 2012 at 10:50 AM, Todd Lipcon <[EMAIL PROTECTED]<javascript:;>
> <javascript:;>>
> >> wrote:
> >> > On Tue, Sep 25, 2012 at 11:21 PM, Konstantin Shvachko
> >> > <[EMAIL PROTECTED]> wrote:
> >> >> I think this is a great work, Todd.
> >> >> And I think we should not merge it into trunk or other branches.
> >> >> As I suggested earlier on this list I think this should be spinned

Best regards,

   - Andy

Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)