Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
HDFS >> mail # dev >> non-local map task input


+
Grandl Robert 2013-07-22, 03:41
+
Grandl Robert 2013-07-22, 17:06
Copy link to this message
-
Re: non-local map task input
Try looking at DFSOutputStream / DFSInputStream.

Sent from a mobile device

Colin
On Jul 21, 2013 8:41 PM, "Grandl Robert" <[EMAIL PROTECTED]> wrote:

> Hi guys,
>
> I am trying to figure out all the points in hdfs code where hdfs traffic
> is read/written. As far as I can tell, it seems most of the traffic goes
> through BlockSender/BlockReceiver, right ?
>
> However, when a client do a copyFromLocal, or read a file, or for a map
> task whose input is not local, it seems the DFSClient is invoked. I
> understand that with DFSClient, it gets the dananodes locations from
> namenode and then directly open a socket and read/writes. Anyway, I am not
> very sure where that happens. Can someone point me out where in the code I
> can find the exact calls to read/write from other datanodes with DFSClient ?
>
> Thanks in advance,
> Robert
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB