Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HDFS, mail # dev - non-local map task input


Copy link to this message
-
Re: non-local map task input
Grandl Robert 2013-07-22, 17:06
Can anyone help me with this please ?

Thanks,
Robert

________________________________
 From: Grandl Robert <[EMAIL PROTECTED]>
To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>
Sent: Sunday, July 21, 2013 8:41 PM
Subject: non-local map task input
 

Hi guys,

I am trying to figure out all the points in hdfs code where hdfs traffic is read/written. As far as I can tell, it seems most of the traffic goes through BlockSender/BlockReceiver, right ?

However, when a client do a copyFromLocal, or read a file, or for a map task whose input is not local, it seems the DFSClient is invoked. I understand that with DFSClient, it gets the dananodes locations from namenode and then directly open a socket and read/writes. Anyway, I am not very sure where that happens. Can someone point me out where in the code I can find the exact calls to read/write from other datanodes with DFSClient ?

Thanks in advance,
Robert