Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop, mail # user - Pragmatic cluster backup strategies?


Copy link to this message
-
Re: Pragmatic cluster backup strategies?
Robert Evans 2012-05-30, 13:36
I am not an expert on the trash so you probably want to verify everything I am about to say.  I believe that trash acts oddly when you try to use it to delete a trash directory.  Quotas can potentially get off when doing this, but I think it still deletes the directory.  Trash is a nice feature, but I wouldn't trust it as a true backup.  I just don't think it is mature enough for something like that.  There are enough issues with quotas that sadly most of our users almost always add -skipTrash all the time.

Where I work we do a combination of several different things depending on the project and their requirements.  In some cases where there are government regulations involved we do regular tape backups.  In other cases we keep the original data around for some time and can re-import it to HDFS if necessary.  In other cases we will copy the data, to multiple Hadoop clusters.  This is usually for the case where we want to do Hot/Warm failover between clusters.  Now we may be different from most other users because we do run lots of different projects on lots of different clusters.

--Bobby Evans

On 5/30/12 1:31 AM, "Darrell Taylor" <[EMAIL PROTECTED]> wrote:

Will "hadoop fs -rm -rf" move everything to the the /trash directory or
will it delete that as well?

I was thinking along the lines of what you suggest, keep the original
source of the data somewhere and then reprocess it all in the event of a
problem.

What do other people do?  Do you run another cluster?  Do you backup
specific parts of the cluster?  Some form of offsite SAN?

On Tue, May 29, 2012 at 6:02 PM, Robert Evans <[EMAIL PROTECTED]> wrote:

> Yes you will have redundancy, so no single point of hardware failure can
> wipe out your data, short of a major catastrophe.  But you can still have
> an errant or malicious "hadoop fs -rm -rf" shut you down.  If you still
> have the original source of your data somewhere else you may be able to
> recover, by reprocessing the data, but if this cluster is your single
> repository for all your data you may have a problem.
>
> --Bobby Evans
>
> On 5/29/12 11:40 AM, "Michael Segel" <[EMAIL PROTECTED]> wrote:
>
> Hi,
> That's not a back up strategy.
> You could still have joe luser take out a key file or directory. What do
> you do then?
>
> On May 29, 2012, at 11:19 AM, Darrell Taylor wrote:
>
> > Hi,
> >
> > We are about to build a 10 machine cluster with 40Tb of storage,
> obviously
> > as this gets full actually trying to create an offsite backup becomes a
> > problem unless we build another 10 machine cluster (too expensive right
> > now).  Not sure if it will help but we have planned the cabinet into an
> > upper and lower half with separate redundant power, then we plan to put
> > half of the cluster in the top, half in the bottom, effectively 2 racks,
> so
> > in theory we could lose half the cluster and still have the copies of all
> > the blocks with a replication factor of 3?  Apart form the data centre
> > burning down or some other disaster that would render the machines
> totally
> > unrecoverable, is this approach good enough?
> >
> > I realise this is a very open question and everyone's circumstances are
> > different, but I'm wondering what other peoples experiences/opinions are
> > for backing up cluster data?
> >
> > Thanks
> > Darrell.
>
>
>