Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # dev >> Need to add fs shim to use QFS


Copy link to this message
-
Re: Need to add fs shim to use QFS
Hi Steve,

Like Harsh said, HADOOP-8886 addresses removing KFS from apache tree.

But I interpret your suggestion as 'moving qfs.jar out of apache tree, and
keeping the jar in possibly a maven repo externally. The new fs shim for
QFS will pull this jar from the repo upon compilation etc.'.
Please correct me if I am wrong.

Thanks,
- Thilee

 

On 10/10/12 12:05 AM, "Harsh J" <[EMAIL PROTECTED]> wrote:

>Hi Steve,
>
>Check out https://issues.apache.org/jira/browse/HADOOP-8886 for the
>KFS removal. Seems relevant to your question here.
>
>On Wed, Oct 10, 2012 at 12:52 AM, Steve Loughran <[EMAIL PROTECTED]>
>wrote:
>> On 5 October 2012 18:27, Thilee Subramaniam <[EMAIL PROTECTED]>
>>wrote:
>>
>>> We at Quantcast have released QFS 1.0 (Quantcast File System) to open
>>> source. This is based on the KFS 0.5 (Kosmos Distributed File System),
>>> a C++ distributed filesystem implementation. KFS plugs into Apache
>>> Hadoop via the 'kfs' shim that is part of Hadoop codebase.
>>>
>>> QFS has added support for permissions, and also, provides fault
>>>tolerance
>>> through Reed-Solomon encoding as well as replication. There are also a
>>> number of performance and stability improvements, including a rewrite
>>>of
>>> the client library to allow parallel concurrent I/Os. Going forward,
>>>new
>>> releases of KFS will come from QFS.
>>>
>>>
>> Does this mean the kfs plugin can go from the apache tree?
>>
>>
>> One problem that we've always had with KFS is that nobody ever tested
>>the
>> filesystem, and it was inevitably out of sync with what was in KFS.
>>
>> Have you considered just pulling the kfs lib out and releasing the
>>bridge
>> classes yourself? It's what the other FS suppliers do, as it gives them
>> more control over the libraries, including the ability to release more
>> often.
>>
>> -steve
>
>
>
>--
>Harsh J
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB