|
|
-
Re: How to add another file system in Hadoop
Hemanth Yamijala 2013-02-21, 12:49
I may be guessing here a bit. Basically a filesystem is identified by the protocol part of the URI of a file - so a file on the S3 filesystem will have a URI like s3://... If you look at the core-default.xml file in Hadoop source, you will see configuration keys like fs.<fs-type>.impl and the value is a class that implements that type of filesystem. So, you'd basically have to define a cdmi:// type protocol and define a fs.cdmi.impl key in your Hadoop configuration.
Thanks hemanth On Thu, Feb 21, 2013 at 4:18 PM, Agarwal, Nikhil <[EMAIL PROTECTED]>wrote:
> Hi,**** > > ** ** > > I am planning to add a file system called CDMI under org.apache.hadoop.fs > in Hadoop, something similar to KFS or S3 which are already there under > org.apache.hadoop.fs. I wanted to ask that say, I write my file system for > CDMI and add the package under fs but then how do I tell the core-site.xml > or other configuration files to use CDMI file system. Where all do I need > to make changes to enable CDMI file system become a part of Hadoop ?**** > > ** ** > > Thanks a lot in advance.**** > > ** ** > > Regards,**** > > Nikhil **** >
+
Hemanth Yamijala 2013-02-21, 12:49
-
Re: How to add another file system in Hadoop
Harsh J 2013-02-21, 12:53
What Hemanth points to (fs.TYPE.impl, i.e. fs.cdmi.impl being set to the classname of the custom FS impl.) is correct for the 1.x releases. In 2.x and ahead, the class for a URI is auto-discovered from the classpath (a 'service'). So as long as your jar is present on the user's runtime, the FS class implementing cdmi:// will get loaded and work without requiring a config injection globally like 1.x did.
On Thu, Feb 21, 2013 at 6:19 PM, Hemanth Yamijala <[EMAIL PROTECTED]> wrote: > I may be guessing here a bit. Basically a filesystem is identified by the > protocol part of the URI of a file - so a file on the S3 filesystem will > have a URI like s3://... If you look at the core-default.xml file in Hadoop > source, you will see configuration keys like fs.<fs-type>.impl and the value > is a class that implements that type of filesystem. So, you'd basically have > to define a cdmi:// type protocol and define a fs.cdmi.impl key in your > Hadoop configuration. > > Thanks > hemanth > > > On Thu, Feb 21, 2013 at 4:18 PM, Agarwal, Nikhil <[EMAIL PROTECTED]> > wrote: >> >> Hi, >> >> >> >> I am planning to add a file system called CDMI under org.apache.hadoop.fs >> in Hadoop, something similar to KFS or S3 which are already there under >> org.apache.hadoop.fs. I wanted to ask that say, I write my file system for >> CDMI and add the package under fs but then how do I tell the core-site.xml >> or other configuration files to use CDMI file system. Where all do I need to >> make changes to enable CDMI file system become a part of Hadoop ? >> >> >> >> Thanks a lot in advance. >> >> >> >> Regards, >> >> Nikhil > >
-- Harsh J
+
Harsh J 2013-02-21, 12:53
|
|