Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HDFS >> mail # dev >> non-local map task input

Copy link to this message
Re: non-local map task input
Try looking at DFSOutputStream / DFSInputStream.

Sent from a mobile device

On Jul 21, 2013 8:41 PM, "Grandl Robert" <[EMAIL PROTECTED]> wrote:

> Hi guys,
> I am trying to figure out all the points in hdfs code where hdfs traffic
> is read/written. As far as I can tell, it seems most of the traffic goes
> through BlockSender/BlockReceiver, right ?
> However, when a client do a copyFromLocal, or read a file, or for a map
> task whose input is not local, it seems the DFSClient is invoked. I
> understand that with DFSClient, it gets the dananodes locations from
> namenode and then directly open a socket and read/writes. Anyway, I am not
> very sure where that happens. Can someone point me out where in the code I
> can find the exact calls to read/write from other datanodes with DFSClient ?
> Thanks in advance,
> Robert