Hi,

We have two clusters but I can't just ssh from one cluster to another - I
have to enter a password, use some authentication, go through a load
balancer etc
It would be very useful for us to use the Zookeeper as a common storage
place for small configuration data, accessible from both these clusters.

So, I'm thinking about starting like 5 servers on each cluster and merge
them into a single quorum - anybody did that?
If yes, how to configure this double-side quorum?

(I'm completely new to the Zookeeper)

Thanks
AY

Aleksey,
  I am a little confused. Whats the motivation behind merging the
quorums? Why do you want 5 on each cluster?

thanks
mahadev

On Tue, Aug 23, 2011 at 10:24 AM, Aleksey Yakovlev
<[EMAIL PROTECTED]> wrote:
> Hi,
>
> We have two clusters but I can't just ssh from one cluster to another - I
> have to enter a password, use some authentication, go through a load
> balancer etc
> It would be very useful for us to use the Zookeeper as a common storage
> place for small configuration data, accessible from both these clusters.
>
> So, I'm thinking about starting like 5 servers on each cluster and merge
> them into a single quorum - anybody did that?
> If yes, how to configure this double-side quorum?
>
> (I'm completely new to the Zookeeper)
>
> Thanks
> AY
>

Mahadev:

I need these two clusters to communicate somehow, because our distributed
system works on both of them - so we need some coordination, for example, to
recover after failures. I could create my own servers/protocols for that,
but why - the ZooKeeper does just that, right?

The number 5 isn't important here, configuration and communication is

Thanks
AY

On Fri, Aug 26, 2011 at 1:55 AM, Mahadev Konar <[EMAIL PROTECTED]>wrote:

> Aleksey,
>  I am a little confused. Whats the motivation behind merging the
> quorums? Why do you want 5 on each cluster?
>
> thanks
> mahadev
>
> On Tue, Aug 23, 2011 at 10:24 AM, Aleksey Yakovlev
> <[EMAIL PROTECTED]> wrote:
> > Hi,
> >
> > We have two clusters but I can't just ssh from one cluster to another - I
> > have to enter a password, use some authentication, go through a load
> > balancer etc
> > It would be very useful for us to use the Zookeeper as a common storage
> > place for small configuration data, accessible from both these clusters.
> >
> > So, I'm thinking about starting like 5 servers on each cluster and merge
> > them into a single quorum - anybody did that?
> > If yes, how to configure this double-side quorum?
> >
> > (I'm completely new to the Zookeeper)
> >
> > Thanks
> > AY
> >
>

As long as the two clusters can ping each other, just set up a single ZK cluster spread across the two, 3 or 5 nodes (total, not per cluster). Note that if these two clusters are used for business continuity purposes (spread across 2 data centers), you still risk the outage of the zookeeper if you lose one of the data centers (the one containing the majority of the ZK nodes).

There's no reason to set up 2 ZK quorums and merge them.

C

-----Original Message-----
From: Aleksey Yakovlev [mailto:[EMAIL PROTECTED]]
Sent: Friday, August 26, 2011 10:09 AM
To: [EMAIL PROTECTED]
Subject: Re: Zookeeper on two clusters?

Mahadev:

I need these two clusters to communicate somehow, because our distributed
system works on both of them - so we need some coordination, for example, to
recover after failures. I could create my own servers/protocols for that,
but why - the ZooKeeper does just that, right?

The number 5 isn't important here, configuration and communication is

Thanks
AY

On Fri, Aug 26, 2011 at 1:55 AM, Mahadev Konar <[EMAIL PROTECTED]>wrote:

> Aleksey,
>  I am a little confused. Whats the motivation behind merging the
> quorums? Why do you want 5 on each cluster?
>
> thanks
> mahadev
>
> On Tue, Aug 23, 2011 at 10:24 AM, Aleksey Yakovlev
> <[EMAIL PROTECTED]> wrote:
> > Hi,
> >
> > We have two clusters but I can't just ssh from one cluster to another - I
> > have to enter a password, use some authentication, go through a load
> > balancer etc
> > It would be very useful for us to use the Zookeeper as a common storage
> > place for small configuration data, accessible from both these clusters.
> >
> > So, I'm thinking about starting like 5 servers on each cluster and merge
> > them into a single quorum - anybody did that?
> > If yes, how to configure this double-side quorum?
> >
> > (I'm completely new to the Zookeeper)
> >
> > Thanks
> > AY
> >
>

Camille:

Sorry to say, I can't ping one cluster from another...
You are saying, that all the servers mutual pingability is necessary and
sufficient for ZK to operate, right?

Thanks
AY

On Fri, Aug 26, 2011 at 12:05 PM, Fournier, Camille F. <
[EMAIL PROTECTED]> wrote:

> As long as the two clusters can ping each other, just set up a single ZK
> cluster spread across the two, 3 or 5 nodes (total, not per cluster). Note
> that if these two clusters are used for business continuity purposes (spread
> across 2 data centers), you still risk the outage of the zookeeper if you
> lose one of the data centers (the one containing the majority of the ZK
> nodes).
>
> There's no reason to set up 2 ZK quorums and merge them.
>
> C
>
> -----Original Message-----
> From: Aleksey Yakovlev [mailto:[EMAIL PROTECTED]]
> Sent: Friday, August 26, 2011 10:09 AM
> To: [EMAIL PROTECTED]
> Subject: Re: Zookeeper on two clusters?
>
> Mahadev:
>
> I need these two clusters to communicate somehow, because our distributed
> system works on both of them - so we need some coordination, for example,
> to
> recover after failures. I could create my own servers/protocols for that,
> but why - the ZooKeeper does just that, right?
>
> The number 5 isn't important here, configuration and communication is
>
> Thanks
> AY
>
> On Fri, Aug 26, 2011 at 1:55 AM, Mahadev Konar <[EMAIL PROTECTED]
> >wrote:
>
> > Aleksey,
> >  I am a little confused. Whats the motivation behind merging the
> > quorums? Why do you want 5 on each cluster?
> >
> > thanks
> > mahadev
> >
> > On Tue, Aug 23, 2011 at 10:24 AM, Aleksey Yakovlev
> > <[EMAIL PROTECTED]> wrote:
> > > Hi,
> > >
> > > We have two clusters but I can't just ssh from one cluster to another -
> I
> > > have to enter a password, use some authentication, go through a load
> > > balancer etc
> > > It would be very useful for us to use the Zookeeper as a common storage
> > > place for small configuration data, accessible from both these
> clusters.
> > >
> > > So, I'm thinking about starting like 5 servers on each cluster and
> merge
> > > them into a single quorum - anybody did that?
> > > If yes, how to configure this double-side quorum?
> > >
> > > (I'm completely new to the Zookeeper)
> > >
> > > Thanks
> > > AY
> > >
> >
>

Not so much mutual pingability, but mutual tcp-ability.

But seriously, why not run ZK in one cluster and put an observer in the
other cluster?  Or do that bi-directionally?

Or put two machines in each cluster and a tie-breaker somewhere outside like
Amazon's EC2?

You haven't provided any real motivation for your original architecture.
 What are you really trying to do? For instance, you say a "distributed
systems works in both centers" and needs "fail-over".  What do you really
mean?

Do you want both data centers to continue to operate in the even of
partition?  In that case you need two separate ZK clusters, possibly with
some cross-datacenter ephemeral files.

Do you want one data center to stop work if it loses contact with the other?
 In that case, put a single ZK cluster in one data-center.

Do you want to ignore most partition issues, but make sure that exactly one
data-center is considered master at all times and that at least one is
considered master as much as possible?  In that case, use the external
tie-breaker architecture.
On Fri, Aug 26, 2011 at 1:39 PM, Aleksey Yakovlev <
[EMAIL PROTECTED]> wrote:

> Camille:
>
> Sorry to say, I can't ping one cluster from another...
> You are saying, that all the servers mutual pingability is necessary and
> sufficient for ZK to operate, right?
>
> Thanks
> AY
>
> On Fri, Aug 26, 2011 at 12:05 PM, Fournier, Camille F. <
> [EMAIL PROTECTED]> wrote:
>
> > As long as the two clusters can ping each other, just set up a single ZK
> > cluster spread across the two, 3 or 5 nodes (total, not per cluster).
> Note
> > that if these two clusters are used for business continuity purposes
> (spread
> > across 2 data centers), you still risk the outage of the zookeeper if you
> > lose one of the data centers (the one containing the majority of the ZK
> > nodes).
> >
> > There's no reason to set up 2 ZK quorums and merge them.
> >
> > C
> >
> > -----Original Message-----
> > From: Aleksey Yakovlev [mailto:[EMAIL PROTECTED]]
> > Sent: Friday, August 26, 2011 10:09 AM
> > To: [EMAIL PROTECTED]
> > Subject: Re: Zookeeper on two clusters?
> >
> > Mahadev:
> >
> > I need these two clusters to communicate somehow, because our distributed
> > system works on both of them - so we need some coordination, for example,
> > to
> > recover after failures. I could create my own servers/protocols for that,
> > but why - the ZooKeeper does just that, right?
> >
> > The number 5 isn't important here, configuration and communication is
> >
> > Thanks
> > AY
> >
> > On Fri, Aug 26, 2011 at 1:55 AM, Mahadev Konar <[EMAIL PROTECTED]
> > >wrote:
> >
> > > Aleksey,
> > >  I am a little confused. Whats the motivation behind merging the
> > > quorums? Why do you want 5 on each cluster?
> > >
> > > thanks
> > > mahadev
> > >
> > > On Tue, Aug 23, 2011 at 10:24 AM, Aleksey Yakovlev
> > > <[EMAIL PROTECTED]> wrote:
> > > > Hi,
> > > >
> > > > We have two clusters but I can't just ssh from one cluster to another
> -
> > I
> > > > have to enter a password, use some authentication, go through a load
> > > > balancer etc
> > > > It would be very useful for us to use the Zookeeper as a common
> storage
> > > > place for small configuration data, accessible from both these
> > clusters.
> > > >
> > > > So, I'm thinking about starting like 5 servers on each cluster and
> > merge
> > > > them into a single quorum - anybody did that?
> > > > If yes, how to configure this double-side quorum?
> > > >
> > > > (I'm completely new to the Zookeeper)
> > > >
> > > > Thanks
> > > > AY
> > > >
> > >
> >
>

Aleksey -

The clusters must be able to communicate over TCP through some user
configurable ports. If this is possible, then as Camille says, you can
configure ZK normally with machines in both locations - no need for any
special quorum setup.

Henry

On 26 August 2011 13:39, Aleksey Yakovlev <[EMAIL PROTECTED]>wrote:

> Camille:
>
> Sorry to say, I can't ping one cluster from another...
> You are saying, that all the servers mutual pingability is necessary and
> sufficient for ZK to operate, right?
>
> Thanks
> AY
>
> On Fri, Aug 26, 2011 at 12:05 PM, Fournier, Camille F. <
> [EMAIL PROTECTED]> wrote:
>
> > As long as the two clusters can ping each other, just set up a single ZK
> > cluster spread across the two, 3 or 5 nodes (total, not per cluster).
> Note
> > that if these two clusters are used for business continuity purposes
> (spread
> > across 2 data centers), you still risk the outage of the zookeeper if you
> > lose one of the data centers (the one containing the majority of the ZK
> > nodes).
> >
> > There's no reason to set up 2 ZK quorums and merge them.
> >
> > C
> >
> > -----Original Message-----
> > From: Aleksey Yakovlev [mailto:[EMAIL PROTECTED]]
> > Sent: Friday, August 26, 2011 10:09 AM
> > To: [EMAIL PROTECTED]
> > Subject: Re: Zookeeper on two clusters?
> >
> > Mahadev:
> >
> > I need these two clusters to communicate somehow, because our distributed
> > system works on both of them - so we need some coordination, for example,
> > to
> > recover after failures. I could create my own servers/protocols for that,
> > but why - the ZooKeeper does just that, right?
> >
> > The number 5 isn't important here, configuration and communication is
> >
> > Thanks
> > AY
> >
> > On Fri, Aug 26, 2011 at 1:55 AM, Mahadev Konar <[EMAIL PROTECTED]
> > >wrote:
> >
> > > Aleksey,
> > >  I am a little confused. Whats the motivation behind merging the
> > > quorums? Why do you want 5 on each cluster?
> > >
> > > thanks
> > > mahadev
> > >
> > > On Tue, Aug 23, 2011 at 10:24 AM, Aleksey Yakovlev
> > > <[EMAIL PROTECTED]> wrote:
> > > > Hi,
> > > >
> > > > We have two clusters but I can't just ssh from one cluster to another
> -
> > I
> > > > have to enter a password, use some authentication, go through a load
> > > > balancer etc
> > > > It would be very useful for us to use the Zookeeper as a common
> storage
> > > > place for small configuration data, accessible from both these
> > clusters.
> > > >
> > > > So, I'm thinking about starting like 5 servers on each cluster and
> > merge
> > > > them into a single quorum - anybody did that?
> > > > If yes, how to configure this double-side quorum?
> > > >
> > > > (I'm completely new to the Zookeeper)
> > > >
> > > > Thanks
> > > > AY
> > > >
> > >
> >
>

--
Henry Robinson
Software Engineer
Cloudera
415-994-6679

Ted:

So, if I find a way to talk to one cluster from another using let's say the
netcat - it'll be enough for ZK, right?

As for the architecture and motivation - there is a lot of people, who
designed them and are managing them, and I'm not one of them. In my opinion
these clusters are too loosely connected, so it's very difficult to build a
distributed software system (which I'm doing) on them.

A simple example: if one cluster sends a file to another, and this another
one gets this file corrupted and needs to ask the first one to resend - it
doesn't have a simple way to do that. Actually, it can send a short file as
a confirmation - but this transaction can also fail. Then the first one will
need to say "I'm waiting for confirmation - where is it?" So, we are talking
about some communication protocol - which is already in the ZK, right?
That's why I'm thinking about the ZK. The second cluster could send the
confirmation to the ZK, and the first one
could receive it from the ZK. And it'd be the ZK internal matter to
replicate the message between its servers.

BTW, what is the "observer" you mentioned?

Thanks
AY

On Fri, Aug 26, 2011 at 5:04 PM, Ted Dunning <[EMAIL PROTECTED]> wrote:

> Not so much mutual pingability, but mutual tcp-ability.
>
> But seriously, why not run ZK in one cluster and put an observer in the

> other cluster?  Or do that bi-directionally?
>
> Or put two machines in each cluster and a tie-breaker somewhere outside
> like
> Amazon's EC2?
>
> You haven't provided any real motivation for your original architecture.
>  What are you really trying to do? For instance, you say a "distributed
> systems works in both centers" and needs "fail-over".  What do you really
> mean?
>
> Do you want both data centers to continue to operate in the even of
> partition?  In that case you need two separate ZK clusters, possibly with
> some cross-datacenter ephemeral files.
>
> Do you want one data center to stop work if it loses contact with the
> other?
>  In that case, put a single ZK cluster in one data-center.
>
> Do you want to ignore most partition issues, but make sure that exactly one
> data-center is considered master at all times and that at least one is
> considered master as much as possible?  In that case, use the external
> tie-breaker architecture.
>
>
> On Fri, Aug 26, 2011 at 1:39 PM, Aleksey Yakovlev <
> [EMAIL PROTECTED]> wrote:
>
> > Camille:
> >
> > Sorry to say, I can't ping one cluster from another...
> > You are saying, that all the servers mutual pingability is necessary and
> > sufficient for ZK to operate, right?
> >
> > Thanks
> > AY
> >
> > On Fri, Aug 26, 2011 at 12:05 PM, Fournier, Camille F. <
> > [EMAIL PROTECTED]> wrote:
> >
> > > As long as the two clusters can ping each other, just set up a single
> ZK
> > > cluster spread across the two, 3 or 5 nodes (total, not per cluster).
> > Note
> > > that if these two clusters are used for business continuity purposes
> > (spread
> > > across 2 data centers), you still risk the outage of the zookeeper if
> you
> > > lose one of the data centers (the one containing the majority of the ZK
> > > nodes).
> > >
> > > There's no reason to set up 2 ZK quorums and merge them.
> > >
> > > C
> > >
> > > -----Original Message-----
> > > From: Aleksey Yakovlev [mailto:[EMAIL PROTECTED]]
> > > Sent: Friday, August 26, 2011 10:09 AM
> > > To: [EMAIL PROTECTED]
> > > Subject: Re: Zookeeper on two clusters?
> > >
> > > Mahadev:
> > >
> > > I need these two clusters to communicate somehow, because our
> distributed
> > > system works on both of them - so we need some coordination, for
> example,
> > > to
> > > recover after failures. I could create my own servers/protocols for
> that,
> > > but why - the ZooKeeper does just that, right?
> > >
> > > The number 5 isn't important here, configuration and communication is
> > >
> > > Thanks
> > > AY
> > >
> > > On Fri, Aug 26, 2011 at 1:55 AM, Mahadev Konar <
> [EMAIL PROTECTED]

Perhaps what you want is more of a messaging solution... Have you looked into, say ActiveMQ or RabbitMQ? You can use ZK for this sort of confirmation stuff but it feels a bit like overkill.

C

-----Original Message-----
From: Aleksey Yakovlev [mailto:[EMAIL PROTECTED]]
Sent: Friday, August 26, 2011 6:06 PM
To: [EMAIL PROTECTED]
Subject: Re: Zookeeper on two clusters?

Ted:

So, if I find a way to talk to one cluster from another using let's say the
netcat - it'll be enough for ZK, right?

As for the architecture and motivation - there is a lot of people, who
designed them and are managing them, and I'm not one of them. In my opinion
these clusters are too loosely connected, so it's very difficult to build a
distributed software system (which I'm doing) on them.

A simple example: if one cluster sends a file to another, and this another
one gets this file corrupted and needs to ask the first one to resend - it
doesn't have a simple way to do that. Actually, it can send a short file as
a confirmation - but this transaction can also fail. Then the first one will
need to say "I'm waiting for confirmation - where is it?" So, we are talking
about some communication protocol - which is already in the ZK, right?
That's why I'm thinking about the ZK. The second cluster could send the
confirmation to the ZK, and the first one
could receive it from the ZK. And it'd be the ZK internal matter to
replicate the message between its servers.

BTW, what is the "observer" you mentioned?

Thanks
AY

On Fri, Aug 26, 2011 at 5:04 PM, Ted Dunning <[EMAIL PROTECTED]> wrote:

> Not so much mutual pingability, but mutual tcp-ability.
>
> But seriously, why not run ZK in one cluster and put an observer in the

> other cluster?  Or do that bi-directionally?
>
> Or put two machines in each cluster and a tie-breaker somewhere outside
> like
> Amazon's EC2?
>
> You haven't provided any real motivation for your original architecture.
>  What are you really trying to do? For instance, you say a "distributed
> systems works in both centers" and needs "fail-over".  What do you really
> mean?
>
> Do you want both data centers to continue to operate in the even of
> partition?  In that case you need two separate ZK clusters, possibly with
> some cross-datacenter ephemeral files.
>
> Do you want one data center to stop work if it loses contact with the
> other?
>  In that case, put a single ZK cluster in one data-center.
>
> Do you want to ignore most partition issues, but make sure that exactly one
> data-center is considered master at all times and that at least one is
> considered master as much as possible?  In that case, use the external
> tie-breaker architecture.
>
>
> On Fri, Aug 26, 2011 at 1:39 PM, Aleksey Yakovlev <
> [EMAIL PROTECTED]> wrote:
>
> > Camille:
> >
> > Sorry to say, I can't ping one cluster from another...
> > You are saying, that all the servers mutual pingability is necessary and
> > sufficient for ZK to operate, right?
> >
> > Thanks
> > AY
> >
> > On Fri, Aug 26, 2011 at 12:05 PM, Fournier, Camille F. <
> > [EMAIL PROTECTED]> wrote:
> >
> > > As long as the two clusters can ping each other, just set up a single
> ZK
> > > cluster spread across the two, 3 or 5 nodes (total, not per cluster).
> > Note
> > > that if these two clusters are used for business continuity purposes
> > (spread
> > > across 2 data centers), you still risk the outage of the zookeeper if
> you
> > > lose one of the data centers (the one containing the majority of the ZK
> > > nodes).
> > >
> > > There's no reason to set up 2 ZK quorums and merge them.
> > >
> > > C
> > >
> > > -----Original Message-----
> > > From: Aleksey Yakovlev [mailto:[EMAIL PROTECTED]]
> > > Sent: Friday, August 26, 2011 10:09 AM
> > > To: [EMAIL PROTECTED]
> > > Subject: Re: Zookeeper on two clusters?
> > >
> > > Mahadev:
> > >
> > > I need these two clusters to communicate somehow, because our
> distributed
> > > system works on both of them - so we need some coordination, for

This is when you set the voting weight of a member of a ZK cluster to zero.
 That means that they cannot help form the quorum and if they disappear, it
doesn't much matter.  They can still be used for read replication and can
forward updates to the master.

This is nice because a secondary data center will always have good and
relatively current data when the link is up, but will never cause a slow
down or cause the master to freeze if the link is down.

On Fri, Aug 26, 2011 at 3:05 PM, Aleksey Yakovlev <
[EMAIL PROTECTED]> wrote:

> BTW, what is the "observer" you mentioned?
>

Camille's comment is exactly correct.  Why not use a messaging system
instead of a coordination system.

Also, if you don't know the exact requirements, then you cannot successfully
use ZK (or a screwdriver) to meet those requirements.

You are correct that loosely coupled distributed systems are very difficult.
 This is related to the fact that with tightly coupled systems you can (sort
of) ignore some of the failure modes for the most part and your system will
still appear to work (much of the time).

The way that I like to say this is that the cost of a Now is largely
determined by its diameter and time scale.  A very large Now costs
exponentially more than a very small Now.  Getting Now on the scale of 5 mm
is pretty cheap.  At 5 m, the cost is significant, but not massive.  At 5000
km, it is very expensive to get a Now better than 50ms.

On Fri, Aug 26, 2011 at 3:05 PM, Aleksey Yakovlev <
[EMAIL PROTECTED]> wrote:

> As for the architecture and motivation - there is a lot of people, who
> designed them and are managing them, and I'm not one of them. In my opinion
> these clusters are too loosely connected, so it's very difficult to build a
> distributed software system (which I'm doing) on them.
>

Ted, Camille:

My example with files sending is just one most evident example. My scripts
need to talk to each other as well. For example, "I'm exiting with code 99,
so all the guys waiting for me - please stop waiting - I'll revive with a
new identity (which I don't know now) - please wait for a signal ABC and you
will know"

I didn't look into any messaging system yet. My biggest problem is
connectivity (load balancers, bastions etc. - please see my starting memo),
so I'm exploring various possibilities trying to understand their
flexibility and requirements to connectivity

BTW, what is "coordination system" exactly? I'd like to see a definition

Thanks
AY

On Fri, Aug 26, 2011 at 6:24 PM, Ted Dunning <[EMAIL PROTECTED]> wrote:

> Camille's comment is exactly correct.  Why not use a messaging system
> instead of a coordination system.
>
> Also, if you don't know the exact requirements, then you cannot
> successfully
> use ZK (or a screwdriver) to meet those requirements.
>
> You are correct that loosely coupled distributed systems are very
> difficult.
>  This is related to the fact that with tightly coupled systems you can
> (sort
> of) ignore some of the failure modes for the most part and your system will
> still appear to work (much of the time).
>
> The way that I like to say this is that the cost of a Now is largely
> determined by its diameter and time scale.  A very large Now costs
> exponentially more than a very small Now.  Getting Now on the scale of 5 mm
> is pretty cheap.  At 5 m, the cost is significant, but not massive.  At
> 5000
> km, it is very expensive to get a Now better than 50ms.
>

On Fri, Aug 26, 2011 at 4:17 PM, Aleksey Yakovlev <
[EMAIL PROTECTED]> wrote:

>
> My example with files sending is just one most evident example. My scripts
> need to talk to each other as well. For example, "I'm exiting with code 99,
> so all the guys waiting for me - please stop waiting - I'll revive with a
> new identity (which I don't know now) - please wait for a signal ABC and
> you
> will know"
>

So for this, you would have a process in data center 1 that maintains an
ephemeral znode in two ZK clusters.  One is in data center 1 and the other
is in data center 2.

Clients of this process would look at this ephemeral znode to determine the
current address of this process.  On partition or process failure, then the
correct ephemerals would disappear.