Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HDFS >> mail # dev >> Integrating Lustre and HDFS


Copy link to this message
-
Re: Integrating Lustre and HDFS
> Your local storage should get used for MR.  Use Lustre via file:// (LocalFileSystem, iirc)
> instead of HDFS via hdfs:// (DistributedFileSystem, irrc) as the default file system type.

If Lustre has integrated checksums, you'll want to use the
RawLocalFileSystem instead of LocalFileSystem. You'll want to make it
accessible via:

fs.raw.impl = org.apache.hadoop.fs.RawLocalFileSystem

so that urls like raw:///my/path won't go through the Hadoop checksum code.

-- Owen
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB