Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop, mail # user - Re: How to add another file system in Hadoop


Copy link to this message
-
Re: How to add another file system in Hadoop
Harsh J 2013-02-21, 12:53
What Hemanth points to (fs.TYPE.impl, i.e. fs.cdmi.impl being set to
the classname of the custom FS impl.) is correct for the 1.x releases.
In 2.x and ahead, the class for a URI is auto-discovered from the
classpath (a 'service'). So as long as your jar is present on the
user's runtime, the FS class implementing cdmi:// will get loaded and
work without requiring a config injection globally like 1.x did.

On Thu, Feb 21, 2013 at 6:19 PM, Hemanth Yamijala
<[EMAIL PROTECTED]> wrote:
> I may be guessing here a bit. Basically a filesystem is identified by the
> protocol part of the URI of a file - so a file on the S3 filesystem will
> have a URI like s3://... If you look at the core-default.xml file in Hadoop
> source, you will see configuration keys like fs.<fs-type>.impl and the value
> is a class that implements that type of filesystem. So, you'd basically have
> to define a cdmi:// type protocol and define a fs.cdmi.impl key in your
> Hadoop configuration.
>
> Thanks
> hemanth
>
>
> On Thu, Feb 21, 2013 at 4:18 PM, Agarwal, Nikhil <[EMAIL PROTECTED]>
> wrote:
>>
>> Hi,
>>
>>
>>
>> I am planning to add a file system called CDMI under org.apache.hadoop.fs
>> in Hadoop, something similar to KFS or S3 which are already there under
>> org.apache.hadoop.fs. I wanted to ask that say, I write my file system for
>> CDMI and add the package under fs but then how do I tell the core-site.xml
>> or other configuration files to use CDMI file system. Where all do I need to
>> make changes to enable CDMI file system become a part of Hadoop ?
>>
>>
>>
>> Thanks a lot in advance.
>>
>>
>>
>> Regards,
>>
>> Nikhil
>
>

--
Harsh J