Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop, mail # user - HDFS interfaces


Copy link to this message
-
RE: HDFS interfaces
John Lilley 2013-06-03, 22:58
Mahmood,

It is the in the FileSystem interface.
http://hadoop.apache.org/docs/current/api/org/apache/hadoop/fs/FileSystem.html#getFileBlockLocations(org.apache.hadoop.fs.Path, long, long)<http://hadoop.apache.org/docs/current/api/org/apache/hadoop/fs/FileSystem.html#getFileBlockLocations(org.apache.hadoop.fs.Path,%20long,%20long)>

This by itself is not sufficient for application programmers to make good use of data locality.  It must be combined with a method of logically splitting the input data along block boundaries, and of launching tasks on worker nodes that are close to the data splits.  MapReduce does both of these things internally along with the file-format input classes.  For an application to do so directly, see the new YARN-based interfaces ApplicationMaster and ResourceManager.  These are however very new and there is little documentation or examples.

john

From: Mahmood Naderan [mailto:[EMAIL PROTECTED]]
Sent: Monday, June 03, 2013 12:09 PM
To: [EMAIL PROTECTED]
Subject: HDFS interfaces

Hello,
It is stated in the "HDFS architecture guide" (https://hadoop.apache.org/docs/r1.0.4/hdfs_design.html) that

HDFS provides interfaces for applications to move themselves closer to where the data is located.

What are these interfaces and where they are in the source code? Is there any manual for the interfaces?

Regards,
Mahmood