Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
HDFS, mail # dev - Integrating Lustre and HDFS


+
Vikas Ashok Patil 2010-06-10, 09:37
+
Allen Wittenauer 2010-06-10, 13:57
Copy link to this message
-
Re: Integrating Lustre and HDFS
Owen O'Malley 2010-06-10, 14:39
> Your local storage should get used for MR.  Use Lustre via file:// (LocalFileSystem, iirc)
> instead of HDFS via hdfs:// (DistributedFileSystem, irrc) as the default file system type.

If Lustre has integrated checksums, you'll want to use the
RawLocalFileSystem instead of LocalFileSystem. You'll want to make it
accessible via:

fs.raw.impl = org.apache.hadoop.fs.RawLocalFileSystem

so that urls like raw:///my/path won't go through the Hadoop checksum code.

-- Owen
+
Vikas Ashok Patil 2010-06-11, 03:27
+
Allen Wittenauer 2010-06-11, 19:02
+
Vikas Ashok Patil 2010-06-12, 03:35
+
Vikas Ashok Patil 2010-06-15, 10:31
+
Allen Wittenauer 2010-06-15, 17:24