-Re: Timeouts and ping handling
Camille Fournier 2012-01-18, 22:03
I think it can be done. Looking through the code, it seems like it should
be safe modulo some stats that are set in the FinalRequestProcessor that
may be less useful.
A question for the other zookeeper devs out there, is there a reason that
we handle read-only operations in the first processor differently on the
leader than the followers? The leader (calling PrepRequestProcessor first)
will do a session check for any of the read-only requests:
but the FollowerRequestProcessor will just push these requests to its
second processor, and never check the session. What's the purpose of the
session check on the leader but not the followers?
On Wed, Jan 18, 2012 at 4:26 PM, Manosiz Bhattacharyya
> We are using Zookeeper-3.3.4 with client session timeouts of 5 seconds,
> and we see frequent timeouts. We have a cluster of 50 nodes (3 of which are
> ZK nodes) and each node has 5 client connections (a total of 250 connection
> to the Ensemble). While investigating the zookeeper connections, we found
> that sometimes pings sent from the zookeeper client does not return from
> the server within 5 seconds, and the client connection gets disconnected.
> Digging deeper it seems that pings are enqueued the same way as other
> requests in the three stage request processing pipeline (prep, sync,
> finalize) in zkserver. So if there are a lot of write operations from other
> active sessions in front of a ping from an inactive session in the queues,
> the inactive session could timeout.
> My question is whether we can return the ping request from the client
> immediately from the server, as the purpose of the ping request seems to be
> to treat it as an heartbeat from relatively inactive sessions. If we keep a
> separate ping queue in the Prep phase which forwards it straight to the
> finalize phase, possible requests before the ping which required I/O inside
> the sync phase would not cause the client timeouts. I hope pings do not
> generate any order in the database. I did take a cursory look at the code
> and thought that could be done. Would really appreciate an opinion
> regarding this.
> As an aside I should mention that increasing the session timeout to 20
> seconds did improved the problem significantly. However as we are using
> Zookeeper to monitor health of our components, increasing the timeout means
> that we only get to know a component's death 20 seconds later. This is
> something we would definitely try to avoid, and would like to go to the 5
> second timeout.