Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
HDFS >> mail # dev >> dfs.datanode.socket.reuse.keepalive


+
Colin McCabe 2013-06-07, 23:29
+
Todd Lipcon 2013-06-10, 17:35
+
Harsh J 2013-06-16, 14:29
+
Colin McCabe 2013-06-17, 18:09
Copy link to this message
-
Re: dfs.datanode.socket.reuse.keepalive
Thanks Colin!

On Mon, Jun 17, 2013 at 11:39 PM, Colin McCabe <[EMAIL PROTECTED]> wrote:
> Thanks for reminding me.  I filed
> https://issues.apache.org/jira/browse/HDFS-4911 for this.
>
> 4307 was about making the cache robust against programs that change
> the wall-clock time.
>
> best,
> Colin
>
>
> On Sun, Jun 16, 2013 at 7:29 AM, Harsh J <[EMAIL PROTECTED]> wrote:
>> Hi Colin,
>>
>> Do we have a JIRA already for this? Is it
>> https://issues.apache.org/jira/browse/HDFS-4307?
>>
>> On Mon, Jun 10, 2013 at 11:05 PM, Todd Lipcon <[EMAIL PROTECTED]> wrote:
>>> +1 for dropping the client side expiry down to something like 1-2 seconds.
>>> I'd rather do that than up the server side, since the server side resource
>>> (DN threads) is likely to be more contended.
>>>
>>> -Todd
>>>
>>> On Fri, Jun 7, 2013 at 4:29 PM, Colin McCabe <[EMAIL PROTECTED]> wrote:
>>>
>>>> Hi all,
>>>>
>>>> HDFS-941 added dfs.datanode.socket.reuse.keepalive.  This allows
>>>> DataXceiver worker threads in the DataNode to linger for a second or
>>>> two after finishing a request, in case the client wants to send
>>>> another request.  On the client side, HDFS-941 added a SocketCache, so
>>>> that subsequent client requests could reuse the same socket.  Sockets
>>>> were closed purely by an LRU eviction policy.
>>>>
>>>> Later, HDFS-3373 added a minimum expiration time to the SocketCache,
>>>> and added a thread which periodically closed old sockets.
>>>>
>>>> However, the default timeout for SocketCache (which is now called
>>>> PeerCache) is much longer than the DN would possibly keep the socket
>>>> open.  Specifically, dfs.client.socketcache.expiryMsec defaults to 2 *
>>>> 60 * 1000 (2 minutes), whereas dfs.datanode.socket.reuse.keepalive
>>>> defaults to 1000 (1 second).
>>>>
>>>> I'm not sure why we have such a big disparity here.  It seems like
>>>> this will inevitably lead to clients trying to use sockets which have
>>>> gone stale, because the server closes them way before the client
>>>> expires them.  Unless I'm missing something, we should probably either
>>>> lengthen the keepalive, or shorten the socket cache expiry, or both.
>>>>
>>>> thoughts?
>>>> Colin
>>>>
>>>
>>>
>>>
>>> --
>>> Todd Lipcon
>>> Software Engineer, Cloudera
>>
>>
>>
>> --
>> Harsh J

--
Harsh J
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB