Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HDFS, mail # dev - Re: [VOTE] Merge HDFS-3077 (QuorumJournalManager) branch to trunk


Copy link to this message
-
Re: [VOTE] Merge HDFS-3077 (QuorumJournalManager) branch to trunk
Eli Collins 2012-10-09, 04:11
On Mon, Oct 8, 2012 at 7:21 PM, Suresh Srinivas <[EMAIL PROTECTED]> wrote:
> On Mon, Oct 8, 2012 at 6:20 PM, Todd Lipcon <[EMAIL PROTECTED]> wrote:
>
>> On Mon, Oct 8, 2012 at 6:01 PM, Suresh Srinivas <[EMAIL PROTECTED]
>> >wrote:
>>
>> > Todd,
>> >
>> > As I indicated in my comments on the jira, I think some of the design
>> > discussions and further simplification of design should happen before the
>> > merge. See -
>> >
>> >
>> https://issues.apache.org/jira/browse/HDFS-3077?focusedCommentId=13470680&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13470680
>> >
>> >
>> I still haven't heard any actual concrete suggestion for a design
>> improvement. Just abstract notions that "it should be simpler". I could say
>> the same about several other features that have been done in the past (e.g
>> federation or YARN) but chose not to because I didn't participate in the
>> development. I generally have faith that, if several other people spent
>> several months working on a project, then there must be good reasons for
>> the complexity that I'm probably overlooking.
>>
>
> I think I have participated enough in this work. Merge time seemed like a
> good time
> to review this more thoroughly given this jira has required quite a
> bit of educating myself and spending a lot of my time on this because of
> need to
> understand the complexities/subtletie how paxos and ZAB applies to the
> namenode
> journals.
>
> Please do not misunderstand that I am trying to block this work from being
> merged. I am
> supportive of this as you have seen in my previous response in this thread.
> All I want
> to see is if we can conclude the discussions and incorporate some the
> comments that come
> up after it.
>
>>
>> Several of us (not just I) spent lots of time on this over the last several
>> months, with all of the work going on in the open. My issue with this
>> discussion happening now is that no one saw fit to raise these questions
>> months ago when the design was first proposed. For example, this most
>> recent question about whether NewEpoch and PrepareRecovery should be
>> separate RPCs could have been raised in late June when the code in question
>> was first introduced as a patch.
>>
>> Nor was anything raised when I gave a "heads up" that the branch was
>> complete and nearly ready for merge ~3 weeks ago. Only once I actually
>> called a merge vote has this discussion started. That doesn't seem like a
>> healthy way to do development.
>>
>
> Again Todd, you are reading more than what is intended. As I said earlier
> merge
> time is a good time to have complete picture and an opportunity to look at
> final design
> and implementation.
>
>
>> What are the criteria, then, for merging? I don't think it's possible to
>> definitively "prove" that a design is "simple". At some level it's a matter
>> of taste. So if the current design works, and is tested, and has people
>> signed up to support it and run it, why not merge? Just like any other part
>> of HDFS, we can continue to _improve_ it after it is in trunk. There are
>> many features that we commit and then improve later on when someone comes
>> up with a way to simplify it.
>>
>
> One concern I have is, once this is merged into trunk I feel that any
> proposed
> improvements may be blocked.

That could be used as justification for not merging any significant
feature.  We've improved HA and federation after they were merged,
there's no reason we won't do the same for QJM.

> Given that why not just wait for these
> discussions
> to complete and do the work in this branch?

We have already delayed it a week per your request, which you agreed
to, and we've already had a vote that passed with three bindings +1s.
That's the protocol of the project - which you should respect. We're
not stopping the merge just because you're afraid some of your
improvements might not be incorporated.

Thanks,
Eli