Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase, mail # user - Setting up NxN replication


Copy link to this message
-
Re: Setting up NxN replication
Demai Ni 2013-11-08, 21:20
hi, Ishan,

can you please elaborate a bit about your use case?

for your example of 3 cluster, I think it should be set up as two M-S for
the same table:family
first one: A = Master, B = Slave
2nd one: B = slave, C = Master.
There is no need to setup the direct relationship between A and C

Demai
On Fri, Nov 8, 2013 at 1:14 PM, Ishan Chhabra <[EMAIL PROTECTED]>wrote:

> I want to setup NxN replication i.e. N clusters each replicating to each
> other. N is expected to be around 10.
>
> On doing some research, I realize it is possible after HBASE-7709 fix, but
> it would lead to much more data flowing in the system. eg.
>
> Lets say we have 3 clusters: A,B and C.
> A new write to A will go to B and then C, and also go to C directly via the
> direct path. This leads to unnecessary network usage and writes to WAL of
> B, that should be avoided. Now imagine this with 10 clusters, it won’t
> scale.
>
> One option is to create a minimum spanning tree joining all the clusters
> and make nodes replicate to their immediate peers in a master-master
> fashion. This is much better than NxN mesh, but still has extra network and
> WAL usage. It also suffers from a failure scenarios where the a single
> cluster going down will pause replication to clusters downstream.
>
> What I really want is that the ReplicationSource should only forward
> WALEdits with cluster-id same as the local cluster-id. This seems like a
> straight forward patch to put in.
>
> Any thoughts on the suggested approach or alternatives?
>
> --
> *Ishan Chhabra *| Rocket Scientist | RocketFuel Inc.
>