Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop, mail # dev - Need to add fs shim to use QFS


Copy link to this message
-
Re: Need to add fs shim to use QFS
Eli Collins 2012-10-10, 19:21
On Wed, Oct 10, 2012 at 12:14 PM, Steve Loughran <[EMAIL PROTECTED]> wrote:
> On 10 October 2012 16:03, Thilee Subramaniam <[EMAIL PROTECTED]> wrote:
>
>> Hi Steve,
>>
>> Like Harsh said, HADOOP-8886 addresses removing KFS from apache tree.
>>
>> But I interpret your suggestion as 'moving qfs.jar out of apache tree, and
>> keeping the jar in possibly a maven repo externally. The new fs shim for
>> QFS will pull this jar from the repo upon compilation etc.'.
>>
>
> I think the main question is if its external, what actually needs to be
> done w.r.t. Hadoop's build itself. Is your goal to include the shim JAR in
> the normal Hadoop releases? As that's what I'm not sure is needed -no other
> non-HDFS filesystem needs that to work with Hadoop. What matters more is to
> hook a local Jenkins server to do the nightly builds and tests of hadoop
> branch-1 and trunk onto your filesystem, test out all the releases, and
> then file bug reports if there have been any regressions.

Good point Steve.  This touches on the larger issue of whether it
makes sense to host FS clients for other file systems in Hadoop
itself.  I agree with what I think you're getting which is - if we can
handle the testing and integration via external dependencies it would
probably be better to have the Hadoop client code live and ship as
part of the other projects since it's more likely to be maintained
there.  Perhaps start a DISCUSS thread on common-dev since this
pertains to other file systems aside from QFS?

Thanks,
Eli