Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # user >> Any samples of how to write a custom FileSystem


Copy link to this message
-
Re: Any samples of how to write a custom FileSystem
Steven,

You could also look at HttpFSFilesystem in the hadoop-httpfs module, it is
quite simple and selfcontained.

Cheers.

Alejandro

On Tue, Jan 31, 2012 at 8:37 PM, Harsh J <[EMAIL PROTECTED]> wrote:

> To write a custom filesystem, extend on the FileSystem class.
>
> Depending on the scheme it is supposed to serve, creating an entry
> "fs.<scheme>.impl" in core-site.xml, and then loading it via the
> FileSystem.get(URI, conf) API will auto load it for you, provided the
> URI you pass has the right scheme.
>
> So supposing I have a FS scheme "foo", I'd register it in core-site.xml as:
>
> <name>fs.foo.impl</name>
> <value>com.myorg.FooFileSystem</value>
>
> And then with a URI object from a path that goes "foo:///mypath/", I'd
> do: FileSystem.get(URI, new Configuration()) to get a FooFileSystem
> instance.
>
> Similarly, if you want to overload the local filesystem with your
> class, override the "fs.file.impl" config with your derivative class,
> and that'd be used in your configuration loaded programs in future.
>
> A good, not-so-complex impl. example to look at generally would be the
> S3FileSystem.
>
> On Wed, Feb 1, 2012 at 9:41 AM, Steve Lewis <[EMAIL PROTECTED]> wrote:
> > Specifically how do I register a Custom FileSystem - any sample code
> >
> > --
> > Steven M. Lewis PhD
> > 4221 105th Ave NE
> > Kirkland, WA 98033
> > 206-384-1340 (cell)
> > Skype lordjoe_com
>
>
>
> --
> Harsh J
> Customer Ops. Engineer
> Cloudera | http://tiny.cloudera.com/about
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB