Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop, mail # user - data and metadata in a different folders


Copy link to this message
-
Re: data and metadata in a different folders
Oleg Ruchovets 2011-03-14, 16:13
basically what we want is to split the data stored by hadoop on regular
disks and put data used for searches on faster ssd disk.
if those two types of data can be separated  then how to define it? if not,
is there anything used by hadoop/hdfs that would benefit from being on a
very fast disk (like cache for example) and is small enough - up to 80gb per
server.

Thanks.
Oleg.

2011/3/14 Ahmed Said Nagy <[EMAIL PROTECTED]>

>
> It holds the directory structure and the file information like permissions
> and location of the file. This is usually held at the name node if u want to
> test that try to move a file that has non zero bytes and see what happens if
> it is not creates then propably it means that the master is unable to write
> on the node check the permissions I faced something very similar.Regards
>
>
>
>
>
>
> > From: [EMAIL PROTECTED]
> > Date: Mon, 14 Mar 2011 15:58:19 +0530
> > Subject: Re: data and metadata in a different folders
> > To: [EMAIL PROTECTED]
> >
> > What type of metadata? Stuff like what's held by sequence files? Is it
> > to be held for each file or per directory/set-of-files?
> >
> > On Mon, Mar 14, 2011 at 1:56 PM, Oleg Ruchovets <[EMAIL PROTECTED]>
> wrote:
> > > Hi ,
> > >   Is it possible to configure hadoop (hdfs) in a such way that meta
> data
> > > and data itself will be in a different folders?
> > >
> > > Actually the questions came from Idea that: it is possible to use 2
> file
> > > system : one for data and one for metadata. We want to get an effect
> that
> > > access to meta data is much more faster then to the data itself. Has
> someone
> > > already  implemented  it or has such experience please advice.
> > >
> > > Thanks in advance
> > > Oleg.
> > >
> >
> >
> >
> > --
> > Harsh J
> > http://harshj.com
>
>