-Re: Integrating Lustre and HDFS
Allen Wittenauer 2010-06-10, 13:57
On Jun 10, 2010, at 2:37 AM, Vikas Ashok Patil wrote:
> In my cluster I have the lustre parallel file system which mainly exposes
> storage over a network. Also there is some local space on each node of the
> cluster. This space is not part of the lustre file system. My hadoop
> installation currently makes use of this local file system. However, I would
> like to make use of the space available through lustre(exposed over the
> network). Hence I was thinking of a way to integrate HDFS and lustre, where
> HDFS would manage the local storage and lustre would provide the storage
> over the network.
> Please let me know your thoughts on this.
Your local storage should get used for MR. Use Lustre via file:// (LocalFileSystem, iirc) instead of HDFS via hdfs:// (DistributedFileSystem, irrc) as the default file system type.