Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HDFS >> mail # user >> Can a jobtracker directly access datanode information (block placement) on namenode?


Copy link to this message
-
Re: Can a jobtracker directly access datanode information (block placement) on namenode?
Hi,

Any HDFS client can request a list of block locations for a given file
path (node-level detail of where blocks are placed for a file), via
the FileSystem#getFileBlockLocations API:
http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/fs/FileSystem.html#getFileBlockLocations(org.apache.hadoop.fs.FileStatus,%20long,%20long)

MR too gets this info via the user's InputFormat#getSplits method, and
schedules with these locations.

On Sat, Jul 21, 2012 at 2:06 AM, Kyungyong Lee <[EMAIL PROTECTED]> wrote:
> Hello all,
>
> I want to get datanode information (related to block placement) that is kept
> at a namenode from a jobtracker. As far as I understand, the jobtracker uses
> the locality-of-data for job scheduling, so I believe the jobtracker is
> keeping the information somewhere in the source code. However, I could not
> find the location. Can anyone give me a starting point (source code) where
> the jobtracker has access to block placement information? Thanks.
>

--
Harsh J
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB