Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HDFS, mail # user - file caching in HDFS ?


Copy link to this message
-
Re: file caching in HDFS ?
Harsh J 2012-04-26, 06:16
Hi Akshay,

You may be interested in the work carried out on
https://issues.apache.org/jira/browse/HADOOP-7714 (For HDFS side, head
to https://issues.apache.org/jira/browse/HDFS-2465 as mentioned on it)

On Thu, Apr 26, 2012 at 11:37 AM, Akshay Singh <[EMAIL PROTECTED]> wrote:
> Hi,
>
> I was looking for caching mechanisms in Hadoop, and was expecting file/block
> caching on Datanodes for frequently accessed file-blocks.
>
> As it seems, HDFS does not provide any caching below the file system
> interface and utilizes DataNode's OS buffer cache for keeping frequently
> accessed HDFS-file (stored as local OS files) in memory. Am i missing
> anything ?
>
> Also, is there any extension to HDFS which has implemented file caching at
> DataNode level ? I understand that this another level of caching would bring
> up issues like data-coherence, but I guess the performance gain may be worth
> paying for this consistency overhead.
>
> P.S. : I am looking for memory based cache on Datanodes, in case it was not
> clear.
>
> Thanks,
> Akshay

--
Harsh J