Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Zookeeper >> mail # user >> Is ZooKeeper the right tool for our needs?


Copy link to this message
-
Re: Is ZooKeeper the right tool for our needs?
This paper from Usenix might shed some light as well, esp around how
things work and why we do what we do (i.e. majority quorums):
http://static.usenix.org/event/usenix10/tech/full_papers/Hunt.pdf

Good Luck,

Patrick

On Mon, Jun 4, 2012 at 3:23 PM, Camille Fournier <[EMAIL PROTECTED]> wrote:
>> However, the
>> ZooKeeper client connects to a ZooKeeper server at random. This is another
>> indication that ZooKeeper might be the wrong tool to use.
>
> You seed the client with the list of servers to connect to. If you
> only seed it with one server, it will only connect to one server even
> if there are others in the quorum, so this should not be an issue.
>
> As for your questions, you're definitely pushing the envelope on the
> standard deployment model but ZK is probably the right tool for you to
> use if you care about correctness in your distributed locking. Using
> ZK for distributed locking is one of its key uses. Embedding it inside
> an application is not common but not unheard of. The danger with
> embedding ZK is that you now have another system that can cause ZK to
> perform poorly due to GC, cpu contention, etc. Is there some reason
> you're concerned with running ZK as a separate VM? It won't be highly
> available, but it sounds like running a standalone ZK on a separate
> server than your two system servers might be the best choice for you
> right now. I would probably recommend this over trying to run an
> embedded quorum.
>
> C
>
> On Mon, Jun 4, 2012 at 5:44 PM, David Nickerson <
> [EMAIL PROTECTED]> wrote:
>
>> I work for a small software company and I have been tasked with researching
>> a distributed lock manager for us to use. I would appreciate any feedback
>> on whether or not ZooKeeper is the right choice.
>>
>> Our software solution utilizes multiple servers, but only the Java
>> application called "Administrator" needs to acquire locks. The server that
>> Administrator runs on is called the System Server, which presents a web
>> interface to potentially thousands of users, and is highly multi-threaded.
>>
>> So far, we have used only one System Server, and we have needed
>> locks only among threads in one application. However, we are looking to
>> scale the number of System Servers, and therefore need a distributed lock
>> manager to be shared among them.
>>
>> We don't want to set up another server just for ZooKeeper, and we don't
>> want the System Server to run another instance of the JVM just for
>> ZooKeeper, so we have decided to embed the ZooKeeper server in our
>> Administrator application.
>>
>> If the System Server goes down for whatever reason, we no longer have a
>> need for locks, so it's not a problem if ZooKeeper goes down too. However,
>> if we have two System Servers, we would like to be able to fail over to the
>> second server. However, ZooKeeper requires that a majority of the ZooKeeper
>> servers be up in order to run. This requirement seems strange and
>> worrisome. For the time being, we can just run an additional ZooKeeper
>> server on another server to give ZooKeeper the redundancy it needs.
>>
>> Another issue is that we would like to have fast locks and reduced network
>> traffic by being able to read from the local ZooKeeper server. However, the
>> ZooKeeper client connects to a ZooKeeper server at random. This is another
>> indication that ZooKeeper might be the wrong tool to use.
>>
>> An important feature that we need is deadlock detection. I had to write
>> this myself, but I was able to do so using the interface that ZooKeeper
>> provides.
>>
>> We also need to be able to administrate the ZooKeeper server from our
>> program. For example, if the ZooKeeper server has an error or we need to
>> shut down the System Server, then the program may need to shutdown the
>> ZooKeeper server or restart it. Some code to do this might be:
>>    // To start the server
>>    QuorumPeerMain main;
>>    try {
>>      QuorumPeerConfig config = new QuorumPeerConfig();