Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase, mail # dev - [PROPOSAL] HBASE-10070 branch


Copy link to this message
-
Re: [PROPOSAL] HBASE-10070 branch
Stack 2014-01-15, 19:17
On Wed, Jan 15, 2014 at 12:44 AM, Enis Söztutar <[EMAIL PROTECTED]>wrote:

> Hi,
>
> I just wanted to give some updates on the HBASE-10070 efforts from the
> technical side, and development side, and propose a branch.
>
> From the technical side:
> The changes for region replicas phase 1 are becoming more mature and
> stable, and most of the "base" changes are starting to become good
> candidates for review. The code has been rebased to trunk, and the main
> working repo has been moved to the HBASE-10070 branch at
> https://github.com/enis/hbase/tree/hbase-10070.
>
> An overview of the changes that is working include:
>  - HRegionInfo & MetaReader & MetaEditor changes for support region
> replicas
>  - HTableDescriptor changes and shell changes for supporting
> REGION_REPLICATION
>  - WebUI changes to display whether a region is a replica or not
>  - AssignmentManager changes coupled with RegionStates & Master changes to
> create and assign replicas, alter table, enable table, etc support.
>
Thanks for the writeup.

I am late to the game so take my comments w/ a grain of salt -- I'll take a
look at HBASE-10070 -- but high-level do we have to go the read replicas
route?  IMO, having our current already-strained AssignmentManager code
base manage three replicas instead of one will ensure that Jimmy Xiang and
Jeffrey Zhong do nothing else for the next year or two but work on the new
interesting use cases introduced by this new level of complexity put upon a
system that has just achieved a hard-won stability.

A few of us chatting offline -- Jimmy, Jon, Elliott, and I -- were
wondering if you couldn't solve this read replicas in a more hbase 'native'
way* by just bringing up three tables -- a main table and then two snapshot
clones with the clones refreshed on a period (via snapshot or via
in-cluster replication) --  and then a shim on top of an HBase client would
read from the main table until failure and then from a snapshot until the
main came back.  Reads from snapshot tables could be marked 'stale'.  You'd
have to modify the balancer so the tables -- or at least their regions --
were physically distinct... you might be able just have the three tables
each in a different namespace.

Or how much more work would it take to follow the route our Facebook
brothers and sisters have taken doing quorum reads and writes incluster?

* When I say 'native' way in the above, what I mean by this is that HBase
has always been about giving clients a 'consistent' view -- at least when
the query is to the source cluster.  Introducing talk and APIs that talk of
'eventual consistency' muddies our story.

> These are some of the remaining things that we are currently working on:
>  - RPC failover support for multi-gets
>  - RPC failover support for scans
>  - RPC cancellation
>

This all sounds great.  I was sort of hoping we wouldn't have to do stuff
like cancellation ourselves though.  Was hoping we could take on an already
done 'rpc' engine that did this kind of stuff for us.

...

> Development side:
> As discussed in the issue design doc
>
> https://issues.apache.org/jira/secure/attachment/12616659/HighAvailabilityDesignforreadsApachedoc.pdf
> "Apache
> code development process" section, at this time we would like to
> propose:
>  (1) Creation of HBASE-10070 branch in svn which will be a fork of trunk as
> of the date branch is created. All of the target authors (me, Devaraj,
> Nicolas, Sergey) are already committers. I do not remember whether our
> bylaws require votes on creating branches.
>

We don't have bylaws.  It is my understanding that any committer can freely
make branches and I see nothing wrong w/ this.

>  (2) The branch will only contain commits that have been reviewed and +1'ed
> from 2 other committers other than the patch author. Every commit in this
> branch will have a single patch (maybe with unforeseen addendums) and and
> associated jira which is a subtask of HBASE-10070.
>

OK.
>  (3) We will use the branch HBASE-10070 hosted at my github repo

All above sounds good.  Let me go look at what is there in HBASE-10070.

St.Ack