Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HDFS >> mail # dev >> Re: [VOTE] Merge HDFS-3077 (QuorumJournalManager) branch to trunk


Copy link to this message
-
Re: [VOTE] Merge HDFS-3077 (QuorumJournalManager) branch to trunk
On Mon, Oct 8, 2012 at 8:03 PM, Andrew Purtell <[EMAIL PROTECTED]> wrote:

> Our position on the QJM is we've already "taken delivery" from the feature
> branch and will maintain a private HDFS fork of branch-2 if necessary, i.e.
> we don't have a significant stake in this discussion except at a meta level
> as potential contributors.
Andrew, as I had said in person to you, thank you for helping in testing
this
feature. This is very useful for the community.
>
> This is a case study of why feature branch development is problematic? A
> would-be contributor can make a significant effort over months and end up
> with a baked result, ready to move on to a perhaps large backlog of other
> work. Others can reasonably be expected meanwhile to triage their attention
> until the merge point. Then, reconsidering design points will be more
> challenging than a design discussion at an earlier time, because there is
> already a significant sunk cost in the current design. What does a feature
> branch buy here over development in situ in trunk (like the BookKeeper JM)?
> Should would-be future contributors consider a feature branch as a viable
> route toward contribution or not?
Andrew, feature branches have worked quite well in HDFS. See many of the
feature developments that have happened in different branches. Let me
reiterate
again. I am not trying to block this work.

I have explained the reasons why merge time made sense for this jira, given
the complexity of design,
where it started with ZAB, then added paxos in the middle of the
development.
These protocols are quite complicated, with multiple papers published on
describing
protocol, the proof, how to implement it and various simplifications and
variations of the protocol.
Requiring intimate knowledge of this and applying it to namenode journals
is non
trivial. I and Sanjay have spent many hours every day last three weeks on
understanding this.
Not all features have this complicated code and protocols and hence do not
require this
level of review.

All I am suggesting here is that we wait for the design discussion to
complete. Based on my comments or Sanjay's comment, you would see that
no one is asking for complete redesign. Some of the subtle issues and their
solutions may not be clear to all. See the discussions in the jira and
compare it with
what is in the design. Second, based on the protocols used, the
questions that are being asked are, can step x and step y be merged. Other
questions
are around perceived diversions from the well known protocols and its
implementation
(example ZooKeeper).

At the end of these discussion there may be few or no changes required.

Regards,
Suresh
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB