Issue with monitoring zookeeper server state using four letter word commands
I came across couple of instances where one zookeeper server was falling
out from the quorum due to some bug/issue with leader election not
We are trying to mitigate this problem by monitoring status of zookeeper
server to check if it is part of the quorum.
If it's not part of the quorum for very long time we restart zookeeper
server so that it can join the quorum again.
Currently there is no way to check if server is part of quorum :
'ruok' returns 'imok' even if zookeeper server is running and is not part
of quorum(i.e it might be continuously running leader election)
'mntr' command reports this information but it doesn't report how long
server is in that state.
I want to restart zookeeper server only if out of quorum for certain amount
of time (say: 2 minutes).
Do I need to add a new four letter word command to report this info or is
there any other way I can achieve this?
I would be more than happy to add this to zookeeper if its helpful for
other zookeeper users.
Thanks & Regards,