Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # user >> Setting up NxN replication


Copy link to this message
-
Setting up NxN replication
I want to setup NxN replication i.e. N clusters each replicating to each
other. N is expected to be around 10.

On doing some research, I realize it is possible after HBASE-7709 fix, but
it would lead to much more data flowing in the system. eg.

Lets say we have 3 clusters: A,B and C.
A new write to A will go to B and then C, and also go to C directly via the
direct path. This leads to unnecessary network usage and writes to WAL of
B, that should be avoided. Now imagine this with 10 clusters, it won’t
scale.

One option is to create a minimum spanning tree joining all the clusters
and make nodes replicate to their immediate peers in a master-master
fashion. This is much better than NxN mesh, but still has extra network and
WAL usage. It also suffers from a failure scenarios where the a single
cluster going down will pause replication to clusters downstream.

What I really want is that the ReplicationSource should only forward
WALEdits with cluster-id same as the local cluster-id. This seems like a
straight forward patch to put in.

Any thoughts on the suggested approach or alternatives?

--
*Ishan Chhabra *| Rocket Scientist | RocketFuel Inc.