Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop, mail # user - Any samples of how to write a custom FileSystem


Copy link to this message
-
Re: Any samples of how to write a custom FileSystem
Alejandro Abdelnur 2012-02-01, 05:20
Steven,

You could also look at HttpFSFilesystem in the hadoop-httpfs module, it is
quite simple and selfcontained.

Cheers.

Alejandro

On Tue, Jan 31, 2012 at 8:37 PM, Harsh J <[EMAIL PROTECTED]> wrote:

> To write a custom filesystem, extend on the FileSystem class.
>
> Depending on the scheme it is supposed to serve, creating an entry
> "fs.<scheme>.impl" in core-site.xml, and then loading it via the
> FileSystem.get(URI, conf) API will auto load it for you, provided the
> URI you pass has the right scheme.
>
> So supposing I have a FS scheme "foo", I'd register it in core-site.xml as:
>
> <name>fs.foo.impl</name>
> <value>com.myorg.FooFileSystem</value>
>
> And then with a URI object from a path that goes "foo:///mypath/", I'd
> do: FileSystem.get(URI, new Configuration()) to get a FooFileSystem
> instance.
>
> Similarly, if you want to overload the local filesystem with your
> class, override the "fs.file.impl" config with your derivative class,
> and that'd be used in your configuration loaded programs in future.
>
> A good, not-so-complex impl. example to look at generally would be the
> S3FileSystem.
>
> On Wed, Feb 1, 2012 at 9:41 AM, Steve Lewis <[EMAIL PROTECTED]> wrote:
> > Specifically how do I register a Custom FileSystem - any sample code
> >
> > --
> > Steven M. Lewis PhD
> > 4221 105th Ave NE
> > Kirkland, WA 98033
> > 206-384-1340 (cell)
> > Skype lordjoe_com
>
>
>
> --
> Harsh J
> Customer Ops. Engineer
> Cloudera | http://tiny.cloudera.com/about
>