Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
HBase, mail # user - Setting up NxN replication


+
Ishan Chhabra 2013-11-08, 21:14
+
Vladimir Rodionov 2013-11-08, 21:20
+
Demai Ni 2013-11-08, 21:36
+
Ishan Chhabra 2013-11-08, 22:16
+
Ishan Chhabra 2013-11-08, 22:29
Copy link to this message
-
Re: Setting up NxN replication
Ted Yu 2013-11-08, 22:33
Ishan:
In your use case, the same table is written to in 10 clusters at roughly
the same time ?

Please clarify.
On Fri, Nov 8, 2013 at 2:29 PM, Ishan Chhabra <[EMAIL PROTECTED]>wrote:

> @Demai,
> We actually have 10 clusters in different locations.
> The replication scope is not an issue for me since I have only one column
> family and we want it replicated to each location.
> Can you elaborate more on why a replication setup of more than 3-4 clusters
> would be a headache in your opinion?
>
>
> On Fri, Nov 8, 2013 at 2:16 PM, Ishan Chhabra <[EMAIL PROTECTED]
> >wrote:
>
> > @Demai,
> > Writes from B should also go to A and C. So, if I were to continue on
> your
> > suggestion, I would setup A-B master master and B-C master-master, which
> is
> > what I was proposing in the 2nd approach (MST based).
> >
> > @Vladimir
> > That is classified. :P
> >
> >
> > On Fri, Nov 8, 2013 at 1:20 PM, Vladimir Rodionov <
> [EMAIL PROTECTED]>wrote:
> >
> >> *I want to setup NxN replication i.e. N clusters each replicating to
> each
> >> other. N is expected to be around 10.*
> >>
> >> Preparing to thermonuclear war?
> >>
> >>
> >>
> >>
> >> On Fri, Nov 8, 2013 at 1:14 PM, Ishan Chhabra <[EMAIL PROTECTED]
> >> >wrote:
> >>
> >> > I want to setup NxN replication i.e. N clusters each replicating to
> each
> >> > other. N is expected to be around 10.
> >> >
> >> > On doing some research, I realize it is possible after HBASE-7709 fix,
> >> but
> >> > it would lead to much more data flowing in the system. eg.
> >> >
> >> > Lets say we have 3 clusters: A,B and C.
> >> > A new write to A will go to B and then C, and also go to C directly
> via
> >> the
> >> > direct path. This leads to unnecessary network usage and writes to WAL
> >> of
> >> > B, that should be avoided. Now imagine this with 10 clusters, it won’t
> >> > scale.
> >> >
> >> > One option is to create a minimum spanning tree joining all the
> clusters
> >> > and make nodes replicate to their immediate peers in a master-master
> >> > fashion. This is much better than NxN mesh, but still has extra
> network
> >> and
> >> > WAL usage. It also suffers from a failure scenarios where the a single
> >> > cluster going down will pause replication to clusters downstream.
> >> >
> >> > What I really want is that the ReplicationSource should only forward
> >> > WALEdits with cluster-id same as the local cluster-id. This seems
> like a
> >> > straight forward patch to put in.
> >> >
> >> > Any thoughts on the suggested approach or alternatives?
> >> >
> >> > --
> >> > *Ishan Chhabra *| Rocket Scientist | RocketFuel Inc.
> >> >
> >>
> >
> >
> >
> > --
> > *Ishan Chhabra *| Rocket Scientist | RocketFuel Inc.
> >
>
>
>
> --
> *Ishan Chhabra *| Rocket Scientist | RocketFuel Inc.
>
+
Ishan Chhabra 2013-11-08, 22:42
+
Demai Ni 2013-11-08, 22:58
+
Ted Yu 2013-11-08, 23:47
+
Demai Ni 2013-11-09, 00:21
+
Ishan Chhabra 2013-11-09, 00:38
+
Demai Ni 2013-11-09, 02:55
+
Ishan Chhabra 2013-11-09, 03:36
+
Demai Ni 2013-11-08, 21:20