Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase, mail # user - Setting up NxN replication


Copy link to this message
-
Re: Setting up NxN replication
Vladimir Rodionov 2013-11-08, 21:20
*I want to setup NxN replication i.e. N clusters each replicating to each
other. N is expected to be around 10.*

Preparing to thermonuclear war?
On Fri, Nov 8, 2013 at 1:14 PM, Ishan Chhabra <[EMAIL PROTECTED]>wrote:

> I want to setup NxN replication i.e. N clusters each replicating to each
> other. N is expected to be around 10.
>
> On doing some research, I realize it is possible after HBASE-7709 fix, but
> it would lead to much more data flowing in the system. eg.
>
> Lets say we have 3 clusters: A,B and C.
> A new write to A will go to B and then C, and also go to C directly via the
> direct path. This leads to unnecessary network usage and writes to WAL of
> B, that should be avoided. Now imagine this with 10 clusters, it won’t
> scale.
>
> One option is to create a minimum spanning tree joining all the clusters
> and make nodes replicate to their immediate peers in a master-master
> fashion. This is much better than NxN mesh, but still has extra network and
> WAL usage. It also suffers from a failure scenarios where the a single
> cluster going down will pause replication to clusters downstream.
>
> What I really want is that the ReplicationSource should only forward
> WALEdits with cluster-id same as the local cluster-id. This seems like a
> straight forward patch to put in.
>
> Any thoughts on the suggested approach or alternatives?
>
> --
> *Ishan Chhabra *| Rocket Scientist | RocketFuel Inc.
>