Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # user >> Setting up NxN replication


Copy link to this message
-
Re: Setting up NxN replication
Ishan:
In your use case, the same table is written to in 10 clusters at roughly
the same time ?

Please clarify.
On Fri, Nov 8, 2013 at 2:29 PM, Ishan Chhabra <[EMAIL PROTECTED]>wrote:

> @Demai,
> We actually have 10 clusters in different locations.
> The replication scope is not an issue for me since I have only one column
> family and we want it replicated to each location.
> Can you elaborate more on why a replication setup of more than 3-4 clusters
> would be a headache in your opinion?
>
>
> On Fri, Nov 8, 2013 at 2:16 PM, Ishan Chhabra <[EMAIL PROTECTED]
> >wrote:
>
> > @Demai,
> > Writes from B should also go to A and C. So, if I were to continue on
> your
> > suggestion, I would setup A-B master master and B-C master-master, which
> is
> > what I was proposing in the 2nd approach (MST based).
> >
> > @Vladimir
> > That is classified. :P
> >
> >
> > On Fri, Nov 8, 2013 at 1:20 PM, Vladimir Rodionov <
> [EMAIL PROTECTED]>wrote:
> >
> >> *I want to setup NxN replication i.e. N clusters each replicating to
> each
> >> other. N is expected to be around 10.*
> >>
> >> Preparing to thermonuclear war?
> >>
> >>
> >>
> >>
> >> On Fri, Nov 8, 2013 at 1:14 PM, Ishan Chhabra <[EMAIL PROTECTED]
> >> >wrote:
> >>
> >> > I want to setup NxN replication i.e. N clusters each replicating to
> each
> >> > other. N is expected to be around 10.
> >> >
> >> > On doing some research, I realize it is possible after HBASE-7709 fix,
> >> but
> >> > it would lead to much more data flowing in the system. eg.
> >> >
> >> > Lets say we have 3 clusters: A,B and C.
> >> > A new write to A will go to B and then C, and also go to C directly
> via
> >> the
> >> > direct path. This leads to unnecessary network usage and writes to WAL
> >> of
> >> > B, that should be avoided. Now imagine this with 10 clusters, it won’t
> >> > scale.
> >> >
> >> > One option is to create a minimum spanning tree joining all the
> clusters
> >> > and make nodes replicate to their immediate peers in a master-master
> >> > fashion. This is much better than NxN mesh, but still has extra
> network
> >> and
> >> > WAL usage. It also suffers from a failure scenarios where the a single
> >> > cluster going down will pause replication to clusters downstream.
> >> >
> >> > What I really want is that the ReplicationSource should only forward
> >> > WALEdits with cluster-id same as the local cluster-id. This seems
> like a
> >> > straight forward patch to put in.
> >> >
> >> > Any thoughts on the suggested approach or alternatives?
> >> >
> >> > --
> >> > *Ishan Chhabra *| Rocket Scientist | RocketFuel Inc.
> >> >
> >>
> >
> >
> >
> > --
> > *Ishan Chhabra *| Rocket Scientist | RocketFuel Inc.
> >
>
>
>
> --
> *Ishan Chhabra *| Rocket Scientist | RocketFuel Inc.
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB