Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
HDFS >> mail # dev >> dfs.datanode.socket.reuse.keepalive


+
Colin McCabe 2013-06-07, 23:29
+
Todd Lipcon 2013-06-10, 17:35
Copy link to this message
-
Re: dfs.datanode.socket.reuse.keepalive
Hi Colin,

Do we have a JIRA already for this? Is it
https://issues.apache.org/jira/browse/HDFS-4307?

On Mon, Jun 10, 2013 at 11:05 PM, Todd Lipcon <[EMAIL PROTECTED]> wrote:
> +1 for dropping the client side expiry down to something like 1-2 seconds.
> I'd rather do that than up the server side, since the server side resource
> (DN threads) is likely to be more contended.
>
> -Todd
>
> On Fri, Jun 7, 2013 at 4:29 PM, Colin McCabe <[EMAIL PROTECTED]> wrote:
>
>> Hi all,
>>
>> HDFS-941 added dfs.datanode.socket.reuse.keepalive.  This allows
>> DataXceiver worker threads in the DataNode to linger for a second or
>> two after finishing a request, in case the client wants to send
>> another request.  On the client side, HDFS-941 added a SocketCache, so
>> that subsequent client requests could reuse the same socket.  Sockets
>> were closed purely by an LRU eviction policy.
>>
>> Later, HDFS-3373 added a minimum expiration time to the SocketCache,
>> and added a thread which periodically closed old sockets.
>>
>> However, the default timeout for SocketCache (which is now called
>> PeerCache) is much longer than the DN would possibly keep the socket
>> open.  Specifically, dfs.client.socketcache.expiryMsec defaults to 2 *
>> 60 * 1000 (2 minutes), whereas dfs.datanode.socket.reuse.keepalive
>> defaults to 1000 (1 second).
>>
>> I'm not sure why we have such a big disparity here.  It seems like
>> this will inevitably lead to clients trying to use sockets which have
>> gone stale, because the server closes them way before the client
>> expires them.  Unless I'm missing something, we should probably either
>> lengthen the keepalive, or shorten the socket cache expiry, or both.
>>
>> thoughts?
>> Colin
>>
>
>
>
> --
> Todd Lipcon
> Software Engineer, Cloudera

--
Harsh J
+
Colin McCabe 2013-06-17, 18:09
+
Harsh J 2013-06-18, 02:50