Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HDFS >> mail # user >> file caching in HDFS ?

Copy link to this message
Re: file caching in HDFS ?
Hi Akshay,

You may be interested in the work carried out on
https://issues.apache.org/jira/browse/HADOOP-7714 (For HDFS side, head
to https://issues.apache.org/jira/browse/HDFS-2465 as mentioned on it)

On Thu, Apr 26, 2012 at 11:37 AM, Akshay Singh <[EMAIL PROTECTED]> wrote:
> Hi,
> I was looking for caching mechanisms in Hadoop, and was expecting file/block
> caching on Datanodes for frequently accessed file-blocks.
> As it seems, HDFS does not provide any caching below the file system
> interface and utilizes DataNode's OS buffer cache for keeping frequently
> accessed HDFS-file (stored as local OS files) in memory. Am i missing
> anything ?
> Also, is there any extension to HDFS which has implemented file caching at
> DataNode level ? I understand that this another level of caching would bring
> up issues like data-coherence, but I guess the performance gain may be worth
> paying for this consistency overhead.
> P.S. : I am looking for memory based cache on Datanodes, in case it was not
> clear.
> Thanks,
> Akshay

Harsh J