Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop, mail # dev - Need to add fs shim to use QFS


Copy link to this message
-
Re: Need to add fs shim to use QFS
Steve Loughran 2012-10-11, 15:01
On 11 October 2012 00:34, Thilee Subramaniam <[EMAIL PROTECTED]> wrote:

>
>
> My initial goal was to make Hadoop use QFS the same way it used KFS. Since
> Hadoop branch-1 had lib/kfs.xx.jar, I was expecting to include a
> qfs.x.x.jar in the Hadoop release; my first patch was to use such jar. But
> now I see that Hadoop trunk links to external maven repos.
>
> It may be reasonable to link qfs.jar from an external source (I haven't
> yet figured out how to serve the maven repo from github for qfs.jar for
> Hadoop - any help on this will be appreciated).

The issue is not so much where qfs.jar comes from as where the
implementation of Hadoop FileSystem goes -the big question being: should it
live in your (OSS) codebase rather than Hadoop's? That would keep it 100%
in sync with your back end, give you the release schedule to suit you, and
ensure that bugreps end up in your bug reporting tools.
> This way your nightly
> builds will work and the tests can catch any qfs related regressions.
>

That regression testing is going to have be on QFS -which means your
infrastructure, real or virtual.