Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
HDFS, mail # dev - questions regarding fuse_dfs_read


+
Aastha Mehta 2011-09-07, 05:45
Copy link to this message
-
Re: questions regarding fuse_dfs_read
Brian Bockelman 2011-09-07, 12:15
Hi Aastha,

A read-ahead buffer is a common technique to trade higher bandwidth for lower latency for a number of common read patterns.  Your OS does something similar (a much more advanced technique though).  By reading ahead, HDFS is betting that your reads have a pattern to it.  I think the 10MB default is a touch excessive (made more sense in previous releases).  I use 32KB.

The buffer is not used if you have very large reads, as it doesn't provide any benefit.

Brian

On Sep 7, 2011, at 12:45 AM, Aastha Mehta wrote:

> Hello,
>
> I am using FUSE-DFS with HDFS for a project. I have to modify the read and
> write functions of fuse_dfs. I have few questions regarding the
> fuse_dfs_read code. There is an rdbuffer_size variable associated with the
> dfs_context, which is by default initialized to 10M. What is this
> rdbuffer_size and what is it used for?
>
> Secondly, in the fuse_dfs_read function, there are two places where
> hdfsPread() is called in a loop. First, there is a check for whether the
> requested read size is greater than the value of rdbuffer_size. Only if it
> is, is the hdfsPread executed. In this case, the data is read into the
> buffer passed from the caller.
>
> In the second case, hdfsPread is executed for if a valid buffer is
> associated with the dfs file handle fh and the size and offset of read
> request lie within the range of the fh->buf. In this case, the data is read
> into fh->buf.
>
> Could someone explain what is happening here?
>
> Thanks,
> Aastha.
>
> --
> Aastha Mehta
> B.E. (Hons.) Computer Science
> BITS Pilani
> E-mail: [EMAIL PROTECTED]

+
Aastha Mehta 2011-09-09, 19:36