Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # user >> Setting up NxN replication


Copy link to this message
-
Re: Setting up NxN replication
Vlad, nice one. :-)

Very good point. Unless your company have data center in 10 different
locations, I don't see a good use case for such complex setup
Also, please keep in mind, currently the only replication type is 'global'.
Before we find a good way to specify peer at table:family level. It will be
a nightmare to replicating in a setup of more than 3~4 clusters

Demai
On Fri, Nov 8, 2013 at 1:20 PM, Vladimir Rodionov <[EMAIL PROTECTED]>wrote:

> *I want to setup NxN replication i.e. N clusters each replicating to each
> other. N is expected to be around 10.*
>
> Preparing to thermonuclear war?
>
>
>
>
> On Fri, Nov 8, 2013 at 1:14 PM, Ishan Chhabra <[EMAIL PROTECTED]
> >wrote:
>
> > I want to setup NxN replication i.e. N clusters each replicating to each
> > other. N is expected to be around 10.
> >
> > On doing some research, I realize it is possible after HBASE-7709 fix,
> but
> > it would lead to much more data flowing in the system. eg.
> >
> > Lets say we have 3 clusters: A,B and C.
> > A new write to A will go to B and then C, and also go to C directly via
> the
> > direct path. This leads to unnecessary network usage and writes to WAL of
> > B, that should be avoided. Now imagine this with 10 clusters, it won’t
> > scale.
> >
> > One option is to create a minimum spanning tree joining all the clusters
> > and make nodes replicate to their immediate peers in a master-master
> > fashion. This is much better than NxN mesh, but still has extra network
> and
> > WAL usage. It also suffers from a failure scenarios where the a single
> > cluster going down will pause replication to clusters downstream.
> >
> > What I really want is that the ReplicationSource should only forward
> > WALEdits with cluster-id same as the local cluster-id. This seems like a
> > straight forward patch to put in.
> >
> > Any thoughts on the suggested approach or alternatives?
> >
> > --
> > *Ishan Chhabra *| Rocket Scientist | RocketFuel Inc.
> >
>