Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Hadoop, mail # dev - Need to add fs shim to use QFS


+
Thilee Subramaniam 2012-10-05, 17:27
+
Steve Loughran 2012-10-09, 19:22
+
Harsh J 2012-10-10, 07:05
+
Thilee Subramaniam 2012-10-10, 15:03
+
Steve Loughran 2012-10-10, 19:14
+
Eli Collins 2012-10-10, 19:21
Copy link to this message
-
Re: Need to add fs shim to use QFS
Steve Loughran 2012-10-11, 08:59
Good point Steve.  This touches on the larger issue of whether it

> makes sense to host FS clients for other file systems in Hadoop
> itself.  I agree with what I think you're getting which is - if we can
> handle the testing and integration via external dependencies it would
> probably be better to have the Hadoop client code live and ship as
> part of the other projects since it's more likely to be maintained
> there.  Perhaps start a DISCUSS thread on common-dev since this
> pertains to other file systems aside from QFS?
>
>
Seems reasonable -I'll let you start it.

We had this problem with Ant; I'm sure the Maven team hit it too: at first
having lots of libraries that bond to external apps makes sense, because
nobody else will do them for you. As your application becomes more
successful, those obscure tasks become a liability as nobody ever
regression tests them, most people don't even have a setup to run them by
hand -and you fear support issues related to them as they will be
non-reproducible, let alone fixable.

Looking at the Ant task list, <netrexxc> and <wljspc> spring to mind -the
latter has only ever been tested in WinNT4 and Solaris 5.x, meaning nobody
has actually run it it since 2001 and Windows XP hitting the market.
http://ant.apache.org/manual/Tasks/wljspc.html

The fact that there are no open JIRAs related to KFS are probably a metric
of its use -again an argument for pushing the work out to the KFS team
-though they will need to work with bigtop to ensure that an RPM can
install the kfs support into /usr/lib/hadoop/lib
+
Thilee Subramaniam 2012-10-10, 23:34
+
Steve Loughran 2012-10-11, 15:01
+
Thilee Subramaniam 2012-10-26, 00:24
+
Steve Loughran 2012-10-26, 17:32
+
Eli Collins 2012-10-05, 17:32
+
Thilee Subramaniam 2012-10-05, 23:27
+
Thilee Subramaniam 2012-10-09, 17:24