I was wondering if someone could clarify watches, was having a tough time following the Java code and the docs seemed.... vague. The docs in question:
"Watches are maintained locally at the ZooKeeper server to which the client is connected. This allows watches to be light weight to set, maintain, and dispatch. When a client connects to a new server, the watch will be triggered for any session events. Watches will not be received while disconnected from a server. When a client reconnects, any previously registered watches will be reregistered and triggered if needed."
1) Watches are fired once, and only once. (Per docs)
2) This states that upon reconnect (but not during disconnect), the watch will be triggered for session events.
3) Upon client reconnect, previous registered watches will be reregistered and triggered (therefore another watch notification?)
Kazoo does these things quite differently, but the end behavior is apparently identical. So, a few questions...
I see from the Java server code that the watches are held at the individual server. So if you connect to a new server but the session has not expired, the watch is obviously not registered there, so it's sent a session event? Which session event?
If you reconnect to the same server, the watch is not triggered at all, unless something happens after its reregistered? If nothing happened, then the watch is not triggered at all even though the connection was lost and re-established?
Is the client expected to reregister the watch if it connects to a new server and the session is still valid?
There's a Zookeeper environment variable that turns auto-reregistration off in the Zookeeper server, why would this exist? Wouldn't it break the expectations of every client?
If the server is holding a watch registry and re-registers it should the client come back before the session expires, why? In a real production setup, most clients will rotate through a list of servers, so it seems remote that with 5+ ZK machines that upon disconnect the client will happen to reconnect to the exact machine it was just at.