Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce >> mail # user >> Has anyone ever written a file system where the data is held in resources


Copy link to this message
-
Re: Has anyone ever written a file system where the data is held in resources
You would probably have to implement your own Hadoop filesystem similar to
S3 and KFS integrate.

I looked at it a while back and it didn't seem insanely difficult …

Kevin

On Wed, Sep 14, 2011 at 9:47 AM, Steve Lewis <[EMAIL PROTECTED]> wrote:

> No - the issue is I want is I want Hadoop to read resources as input files
> as if they were in hdfs - I know how to read resources as
> input streams but I don't know how to get a Hadoop file system which will
> treat a Path like res://myclass/myresource.txt as useful and
> give me an FSInputStream (rather that a simple InputStream)
>
>
> On Wed, Sep 14, 2011 at 9:38 AM, Kevin Burton <[EMAIL PROTECTED]> wrote:
>
>> You can already do this with the JAR file format… if you load a resource
>> via path it uses the class loader system to find it in all available jars.
>>
>> Kevin
>>
>>
>> On Wed, Sep 14, 2011 at 9:24 AM, Steve Lewis <[EMAIL PROTECTED]>wrote:
>>
>>> When writing tests it is useful to keep all data in resources since this
>>> makes automatic execution easier.
>>> The structure of a set of resources should make it easy to have a schema
>>> such as res:// to look there -
>>>  Has anyone already done the work.
>>>
>>> --
>>> Steven M. Lewis PhD
>>> 4221 105th Ave NE
>>> Kirkland, WA 98033
>>> 206-384-1340 (cell)
>>> Skype lordjoe_com
>>>
>>>
>>>
>>
>>
>> --
>>
>> Founder/CEO Spinn3r.com
>>
>> Location: *San Francisco, CA*
>> Skype: *burtonator*
>>
>> Skype-in: *(415) 871-0687*
>>
>>
>
>
> --
> Steven M. Lewis PhD
> 4221 105th Ave NE
> Kirkland, WA 98033
> 206-384-1340 (cell)
> Skype lordjoe_com
>
>
>
--

Founder/CEO Spinn3r.com

Location: *San Francisco, CA*
Skype: *burtonator*

Skype-in: *(415) 871-0687*
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB