Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop, mail # dev - Need to add fs shim to use QFS


Copy link to this message
-
Need to add fs shim to use QFS
Thilee Subramaniam 2012-10-05, 17:27
We at Quantcast have released QFS 1.0 (Quantcast File System) to open
source. This is based on the KFS 0.5 (Kosmos Distributed File System),
a C++ distributed filesystem implementation. KFS plugs into Apache
Hadoop via the 'kfs' shim that is part of Hadoop codebase.

QFS has added support for permissions, and also, provides fault tolerance
through Reed-Solomon encoding as well as replication. There are also a
number of performance and stability improvements, including a rewrite of
the client library to allow parallel concurrent I/Os. Going forward, new
releases of KFS will come from QFS.

The open source release of QFS is at http://quantcast.github.com/qfs

QFS plugs into Apache Hadoop the same way KFS does. Currently, one would
apply the patches or JARs from the QFS source tree onto Apache Hadoop to
make Hadoop use QFS. The patch for Apache Hadoop 1.0.X can be found at
https://github.com/quantcast/qfs/blob/master/hadoop/hadoop-1.0.X.patch

In order to make the integration seamless, we would like to add a 'qfs'
shim to Apache Hadoop so that the current active branches (1.0.X, 2.X.X,
0.23.X) of Apache Hadoop can use QFS.

Towards this, I've submited an ASF JIRA feature ticket (HADOOP-8885) under
hadoop-common project, and send a pull request with the QFS shim changes
to https://github.com/apache/hadoop-common/tree/branch-1.0.2

I will subsequently submit pull requests to the other active Hadoop
branches.

If you have any question, I will be happy yo answer or provide more
details on QFS.

- Thilee