-Re: file caching in HDFS ?
Harsh J 2012-04-26, 06:16
You may be interested in the work carried out on
https://issues.apache.org/jira/browse/HADOOP-7714 (For HDFS side, head
to https://issues.apache.org/jira/browse/HDFS-2465 as mentioned on it)
On Thu, Apr 26, 2012 at 11:37 AM, Akshay Singh <[EMAIL PROTECTED]> wrote:
> I was looking for caching mechanisms in Hadoop, and was expecting file/block
> caching on Datanodes for frequently accessed file-blocks.
> As it seems, HDFS does not provide any caching below the file system
> interface and utilizes DataNode's OS buffer cache for keeping frequently
> accessed HDFS-file (stored as local OS files) in memory. Am i missing
> anything ?
> Also, is there any extension to HDFS which has implemented file caching at
> DataNode level ? I understand that this another level of caching would bring
> up issues like data-coherence, but I guess the performance gain may be worth
> paying for this consistency overhead.
> P.S. : I am looking for memory based cache on Datanodes, in case it was not