Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
HDFS >> mail # dev >> Integrating Lustre and HDFS

Vikas Ashok Patil 2010-06-10, 09:37
Copy link to this message
Re: Integrating Lustre and HDFS

On Jun 10, 2010, at 2:37 AM, Vikas Ashok Patil wrote:
> In my cluster I have the lustre parallel file system which mainly exposes
> storage over a network. Also there is some local space on each node of the
> cluster. This space is not part of the lustre file system. My hadoop
> installation currently makes use of this local file system. However, I would
> like to make use of the space available through lustre(exposed over the
> network). Hence I was thinking of a way to integrate HDFS and lustre, where
> HDFS would manage the local storage and lustre would provide the storage
> over the network.
> Please let me know your thoughts on this.

Your local storage should get used for MR.  Use Lustre via file:// (LocalFileSystem, iirc) instead of HDFS via hdfs:// (DistributedFileSystem, irrc) as the default file system type.  
Owen OMalley 2010-06-10, 14:39
Vikas Ashok Patil 2010-06-11, 03:27
Allen Wittenauer 2010-06-11, 19:02
Vikas Ashok Patil 2010-06-12, 03:35
Vikas Ashok Patil 2010-06-15, 10:31
Allen Wittenauer 2010-06-15, 17:24