Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Zookeeper, mail # user - Distributed ZooKeeper cluster design


+
Dima Gutzeit 2011-12-13, 11:29
+
Ted Dunning 2011-12-13, 14:05
+
Camille Fournier 2011-12-13, 15:44
+
Camille Fournier 2011-12-13, 15:50
+
Dima Gutzeit 2011-12-13, 16:09
+
Ted Dunning 2011-12-13, 16:23
+
Ted Dunning 2011-12-13, 16:24
+
Camille Fournier 2011-12-13, 16:24
Copy link to this message
-
Re: Distributed ZooKeeper cluster design
Benjamin Reed 2011-12-13, 19:57
i agree with camille that mirror breaks a lot of the basic guarantees
that you use from zookeeper. with that caveat in mind, there is a
patch that enables mirroring: ZOOKEEPER-892.

ben

On Tue, Dec 13, 2011 at 8:24 AM, Camille Fournier <[EMAIL PROTECTED]> wrote:
> I have to strongly disagree with ted on the mirroring idea... I think it is
> likely to be really error-prone and kind of defeats the purpose of ZK in my
> mind. It depends on what you're mirroring but if you're trying to keep all
> the data coherent you can't sensibly do that in two clusters, so unless the
> mirror is for a really small subset of the data I would stay far far away
> from that.
>
> Observers are available in 3.3.3, yes.
> Unfortunately, we don't have configurable connection logic in ZK client (at
> least java) right now. We have the ability to add it pretty easily, but it
> hasn't been put in yet.
>
> You're seeing slow performance for a setup that has all ZK servers in
> region C for clients only in regions A and B and you can't blame the
> network? That's literally the only thing you could blame, unless clients in
> region C were also seeing slow performance or they have some other problem
> in they way they are implemented that makes them different from clients
> running in region C.
>
> C
>
> On Tue, Dec 13, 2011 at 11:09 AM, Dima Gutzeit
> <[EMAIL PROTECTED]>wrote:
>
>> Ted and Camille,
>>
>> Thanks for a very details response.
>>
>> At the moment I have an option A implemented in production and what I see
>> is that ZK client in A and B have a "slow" performance (even reads) and I
>> can't really blame the network since it does not look like a real
>> bottleneck.
>>
>> I wonder if doing option 2 will improve the ZK client performance/speed ...
>>
>> As for my use case, its around 50/50 reads and writes.
>>
>> As for fallback, ofcourse in A and B I would want to define C as a backup,
>> not sure how it can be done since as I understand if I supply several
>> addresses in the connection string the client will use one, randomly.
>>
>> About Ted's suggestion to consider having several clusters and to have a
>> special process to mirror, is it something available as part of ZooKeeper ?
>>
>> I also read about observers (is it available in 3.3.3 ?) and it seems to be
>> a good option is my case, which brings me to the question of how to
>> configure explicit fallback instead of random client selection ? If I want
>> to tell ZK client in B to use the local B instance (observer) and if it
>> fails then contact ANY server in the C (with a list of several).
>>
>> Thanks in advance.
>>
>> Regards,
>> Dima Gutzeit.
>>
>>
>>
>> On Tue, Dec 13, 2011 at 5:44 PM, Camille Fournier <[EMAIL PROTECTED]
>> >wrote:
>>
>> > Ted is of course right, but to speculate:
>> >
>> > The idea you had with 3 in C, one in A and one in B isn't bad, given
>> > some caveats.
>> >
>> > With 3 in C, as long as they are all available, quorum should live in
>> > C and you shouldn't have much slowdown from the remote servers in A
>> > and B. However, if you point your A servers only to the A zookeeper,
>> > you have a failover risk where your A servers will have no ZK if the
>> > sever in region A goes down (same with B, of course). If you have a
>> > lot of servers in the outer regions, this could be a risk. You are
>> > also giving up any kind of load balancing for the A and B region ZKs,
>> > which may not be important but is good to know.
>> >
>> > Another thing to be aware of is that the A and B region ZKs will have
>> > slower write response time due to the WAN cost, and they will tend to
>> > lag behind the majority cluster a bit. This shouldn't cause
>> > correctness issues but could impact client performance in those
>> > regions.
>> >
>> > Honestly, if you're doing a read-mostly workload in the A and B
>> > regions, I doubt this is a bad design. It's pretty easy to test ZK
>> > setups using Pat's zksmoketest utility, so you might try setting up
>> > the sample cluster and running some of the smoketests on it.
+
Ted Dunning 2011-12-14, 00:36
+
Dima Gutzeit 2011-12-14, 02:50
+
Yongsheng Wu 2011-12-14, 23:04
+
Camille Fournier 2011-12-15, 01:41
+
Henry Robinson 2011-12-13, 18:37