I am wondering if QoS on handler threads is being looked at by someone ? We
have found huge gains by doing this for asymmetric workloads which are
extremely write heavy but latency on reads matters the most. Basically we
forked out a separate thread pool of X threads for handling reads to
prevent starvation of read requests which were far outnumbered by the # of
As of now, we have been making adhoc changes (like introduce higher
priority levels for certain types of requests like gets and scans) but
having a framework to control this would be a really nice thing.
One simple way to achieve this would be to have a group of "high priority
handler threads" and be able to mark each table to prioritize
"reads/writes". On receiving an RPC, the request is directly dispatched to
this other pool of threads and the remaining RPCs go to the regular pool.
This would be a naive implementation.
The issue with this could be that what happens when the user ends up
directing most of their load to the high priority pool which has fewer
threads. We could do something simple like having a tight upper bound on
the call queue length and if a new high priority call is rejected from this
pool, just enqueue it to the regular pool of requests.