Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase, mail # user - Setting up NxN replication


Copy link to this message
-
Re: Setting up NxN replication
Ishan Chhabra 2013-11-08, 22:29
@Demai,
We actually have 10 clusters in different locations.
The replication scope is not an issue for me since I have only one column
family and we want it replicated to each location.
Can you elaborate more on why a replication setup of more than 3-4 clusters
would be a headache in your opinion?
On Fri, Nov 8, 2013 at 2:16 PM, Ishan Chhabra <[EMAIL PROTECTED]>wrote:

> @Demai,
> Writes from B should also go to A and C. So, if I were to continue on your
> suggestion, I would setup A-B master master and B-C master-master, which is
> what I was proposing in the 2nd approach (MST based).
>
> @Vladimir
> That is classified. :P
>
>
> On Fri, Nov 8, 2013 at 1:20 PM, Vladimir Rodionov <[EMAIL PROTECTED]>wrote:
>
>> *I want to setup NxN replication i.e. N clusters each replicating to each
>> other. N is expected to be around 10.*
>>
>> Preparing to thermonuclear war?
>>
>>
>>
>>
>> On Fri, Nov 8, 2013 at 1:14 PM, Ishan Chhabra <[EMAIL PROTECTED]
>> >wrote:
>>
>> > I want to setup NxN replication i.e. N clusters each replicating to each
>> > other. N is expected to be around 10.
>> >
>> > On doing some research, I realize it is possible after HBASE-7709 fix,
>> but
>> > it would lead to much more data flowing in the system. eg.
>> >
>> > Lets say we have 3 clusters: A,B and C.
>> > A new write to A will go to B and then C, and also go to C directly via
>> the
>> > direct path. This leads to unnecessary network usage and writes to WAL
>> of
>> > B, that should be avoided. Now imagine this with 10 clusters, it won’t
>> > scale.
>> >
>> > One option is to create a minimum spanning tree joining all the clusters
>> > and make nodes replicate to their immediate peers in a master-master
>> > fashion. This is much better than NxN mesh, but still has extra network
>> and
>> > WAL usage. It also suffers from a failure scenarios where the a single
>> > cluster going down will pause replication to clusters downstream.
>> >
>> > What I really want is that the ReplicationSource should only forward
>> > WALEdits with cluster-id same as the local cluster-id. This seems like a
>> > straight forward patch to put in.
>> >
>> > Any thoughts on the suggested approach or alternatives?
>> >
>> > --
>> > *Ishan Chhabra *| Rocket Scientist | RocketFuel Inc.
>> >
>>
>
>
>
> --
> *Ishan Chhabra *| Rocket Scientist | RocketFuel Inc.
>

--
*Ishan Chhabra *| Rocket Scientist | RocketFuel Inc.