Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HDFS >> mail # dev >> Integrating Lustre and HDFS

Copy link to this message
Re: Integrating Lustre and HDFS

On Jun 10, 2010, at 2:37 AM, Vikas Ashok Patil wrote:
> In my cluster I have the lustre parallel file system which mainly exposes
> storage over a network. Also there is some local space on each node of the
> cluster. This space is not part of the lustre file system. My hadoop
> installation currently makes use of this local file system. However, I would
> like to make use of the space available through lustre(exposed over the
> network). Hence I was thinking of a way to integrate HDFS and lustre, where
> HDFS would manage the local storage and lustre would provide the storage
> over the network.
> Please let me know your thoughts on this.

Your local storage should get used for MR.  Use Lustre via file:// (LocalFileSystem, iirc) instead of HDFS via hdfs:// (DistributedFileSystem, irrc) as the default file system type.