Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Hadoop, mail # dev - short-circuiting HDFS reads


+
Jun Rao 2009-01-07, 17:19
+
Doug Cutting 2009-01-07, 18:18
+
Raghu Angadi 2009-01-07, 18:18
+
George Porter 2009-01-08, 18:13
+
Chris K Wensel 2009-01-08, 18:25
+
Doug Cutting 2009-01-08, 18:39
+
stack 2009-01-08, 22:23
+
Jun Rao 2009-01-12, 04:16
+
George Porter 2009-01-12, 18:11
+
Jun Rao 2009-01-13, 03:30
Copy link to this message
-
Re: short-circuiting HDFS reads
Sanjay Radia 2009-02-13, 22:39

On Jan 8, 2009, at 10:13 AM, George Porter wrote:

> Hi Jun,
>
> The earlier responses to your email reference the JIRA that I opened
> about this issue.  Short-circuiting the primary HDFS datapath does
> improve throughput, and the amount depends on your workload (random
> reads especially).  Some initial experimental results are posted to  
> that
> JIRA.  A second advantage is that since the JVM hosting the HDFS  
> client
> is doing the reading, the O/S will satisfy future disk requests from  
> the
> cache, which isn't really possible when you read over the network  
> (even
> to another JVM on the same host).
>
> There are several real disadvantages, the largest of which include  
> 1) it
> adds a new datapath, and 2) bypasses various security and auditing
> features of HDFS.
>
We are in middle of adding security to HDFS.
Having the client read the blocks directly would violate security.  
Security is a specially thorny problem to solve in this case.
Further the internal structure and hence the path name of the file are  
not visible outside.
One could consider hacking this (ignoring security) but even this gets  
tricky as the directory in which the block is saved may change if
some one starts to write to the file (which  can happen with the  
recent  append work),

Interesting optimization but tricky to do in a clean way (at least not  
obvious to me).
sanjay
> I would certainly like to think through a more clean
> interface for achieving this goal, especially since reading local data
> should be the common case.  Any thoughts you might have would be
> appreciated.
>
> Thanks,
> George
>
> Jun Rao wrote:
> > Hi,
> >
> > Today, HDFS always reads through a socket even when the data is  
> local to
> > the client. This adds a lot of overhead, especially for warm  
> reads. It
> > should be possible for a dfs client to test if a block to be read  
> is local
> > and if so, bypass socket and read through local FS api directly.  
> This
> > should improve random access performance significantly (e.g., for  
> HBase).
> > Has this been considered in HDFS? Thanks,
> >
> > Jun
> >
> >
>

+
Dhruba Borthakur 2009-02-14, 05:45