Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HDFS >> mail # dev >> Integrating Lustre and HDFS

Copy link to this message
Re: Integrating Lustre and HDFS
> Your local storage should get used for MR.  Use Lustre via file:// (LocalFileSystem, iirc)
> instead of HDFS via hdfs:// (DistributedFileSystem, irrc) as the default file system type.

If Lustre has integrated checksums, you'll want to use the
RawLocalFileSystem instead of LocalFileSystem. You'll want to make it
accessible via:

fs.raw.impl = org.apache.hadoop.fs.RawLocalFileSystem

so that urls like raw:///my/path won't go through the Hadoop checksum code.

-- Owen