Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Zookeeper >> mail # user >> Distributed ZooKeeper cluster design


Copy link to this message
-
Re: Distributed ZooKeeper cluster design
I am happy to agree with everyone that the mirroring isn't a great idea for
most things even if that makes me look like I disagree with myself.

I do think that mirroring could be made to happen in a reliable way, but it
isn't going to be a viable substitute for direct access to the cluster.  By
reliable, I think that you could get a reliable picture of what was in the
master cluster at some time in the past.  Occasionally the mirror would be
further behind than other times and it might be necessary for the mirror to
be updated much faster than real-time.  In my vision, the mirror would be
read-only since anything else leads to madness in the strict consistency
model that ZK maintains.

On Tue, Dec 13, 2011 at 2:57 PM, Benjamin Reed <[EMAIL PROTECTED]> wrote:

> i agree with camille that mirror breaks a lot of the basic guarantees
> that you use from zookeeper. with that caveat in mind, there is a
> patch that enables mirroring: ZOOKEEPER-892.
>
> ben
>
> On Tue, Dec 13, 2011 at 8:24 AM, Camille Fournier <[EMAIL PROTECTED]>
> wrote:
> > I have to strongly disagree with ted on the mirroring idea... I think it
> is
> > likely to be really error-prone and kind of defeats the purpose of ZK in
> my
> > mind. It depends on what you're mirroring but if you're trying to keep
> all
> > the data coherent you can't sensibly do that in two clusters, so unless
> the
> > mirror is for a really small subset of the data I would stay far far away
> > from that.
> >
> > Observers are available in 3.3.3, yes.
> > Unfortunately, we don't have configurable connection logic in ZK client
> (at
> > least java) right now. We have the ability to add it pretty easily, but
> it
> > hasn't been put in yet.
> >
> > You're seeing slow performance for a setup that has all ZK servers in
> > region C for clients only in regions A and B and you can't blame the
> > network? That's literally the only thing you could blame, unless clients
> in
> > region C were also seeing slow performance or they have some other
> problem
> > in they way they are implemented that makes them different from clients
> > running in region C.
> >
> > C
> >
> > On Tue, Dec 13, 2011 at 11:09 AM, Dima Gutzeit
> > <[EMAIL PROTECTED]>wrote:
> >
> >> Ted and Camille,
> >>
> >> Thanks for a very details response.
> >>
> >> At the moment I have an option A implemented in production and what I
> see
> >> is that ZK client in A and B have a "slow" performance (even reads) and
> I
> >> can't really blame the network since it does not look like a real
> >> bottleneck.
> >>
> >> I wonder if doing option 2 will improve the ZK client performance/speed
> ...
> >>
> >> As for my use case, its around 50/50 reads and writes.
> >>
> >> As for fallback, ofcourse in A and B I would want to define C as a
> backup,
> >> not sure how it can be done since as I understand if I supply several
> >> addresses in the connection string the client will use one, randomly.
> >>
> >> About Ted's suggestion to consider having several clusters and to have a
> >> special process to mirror, is it something available as part of
> ZooKeeper ?
> >>
> >> I also read about observers (is it available in 3.3.3 ?) and it seems
> to be
> >> a good option is my case, which brings me to the question of how to
> >> configure explicit fallback instead of random client selection ? If I
> want
> >> to tell ZK client in B to use the local B instance (observer) and if it
> >> fails then contact ANY server in the C (with a list of several).
> >>
> >> Thanks in advance.
> >>
> >> Regards,
> >> Dima Gutzeit.
> >>
> >>
> >>
> >> On Tue, Dec 13, 2011 at 5:44 PM, Camille Fournier <[EMAIL PROTECTED]
> >> >wrote:
> >>
> >> > Ted is of course right, but to speculate:
> >> >
> >> > The idea you had with 3 in C, one in A and one in B isn't bad, given
> >> > some caveats.
> >> >
> >> > With 3 in C, as long as they are all available, quorum should live in
> >> > C and you shouldn't have much slowdown from the remote servers in A
> >> > and B. However, if you point your A servers only to the A zookeeper,
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB