Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HDFS >> mail # user >> file caching in HDFS ?


Copy link to this message
-
Re: file caching in HDFS ?
Hi Akshay,

You may be interested in the work carried out on
https://issues.apache.org/jira/browse/HADOOP-7714 (For HDFS side, head
to https://issues.apache.org/jira/browse/HDFS-2465 as mentioned on it)

On Thu, Apr 26, 2012 at 11:37 AM, Akshay Singh <[EMAIL PROTECTED]> wrote:
> Hi,
>
> I was looking for caching mechanisms in Hadoop, and was expecting file/block
> caching on Datanodes for frequently accessed file-blocks.
>
> As it seems, HDFS does not provide any caching below the file system
> interface and utilizes DataNode's OS buffer cache for keeping frequently
> accessed HDFS-file (stored as local OS files) in memory. Am i missing
> anything ?
>
> Also, is there any extension to HDFS which has implemented file caching at
> DataNode level ? I understand that this another level of caching would bring
> up issues like data-coherence, but I guess the performance gain may be worth
> paying for this consistency overhead.
>
> P.S. : I am looking for memory based cache on Datanodes, in case it was not
> clear.
>
> Thanks,
> Akshay

--
Harsh J
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB