HDFS-941 added dfs.datanode.socket.reuse.keepalive. This allows
DataXceiver worker threads in the DataNode to linger for a second or
two after finishing a request, in case the client wants to send
another request. On the client side, HDFS-941 added a SocketCache, so
that subsequent client requests could reuse the same socket. Sockets
were closed purely by an LRU eviction policy.
Later, HDFS-3373 added a minimum expiration time to the SocketCache,
and added a thread which periodically closed old sockets.
However, the default timeout for SocketCache (which is now called
PeerCache) is much longer than the DN would possibly keep the socket
open. Specifically, dfs.client.socketcache.expiryMsec defaults to 2 *
60 * 1000 (2 minutes), whereas dfs.datanode.socket.reuse.keepalive
defaults to 1000 (1 second).
I'm not sure why we have such a big disparity here. It seems like
this will inevitably lead to clients trying to use sockets which have
gone stale, because the server closes them way before the client
expires them. Unless I'm missing something, we should probably either
lengthen the keepalive, or shorten the socket cache expiry, or both.