Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HDFS, mail # dev - Heads up - Snapshots feature merge into trunk


Copy link to this message
-
Re: Heads up - Snapshots feature merge into trunk
Tsz Wo Sze 2013-04-18, 18:48
Hi Aaron,

Thanks for supporting the snapshot feature.

Currently, allowSnapshot(..) and disallowSnapshot(..) are already in HdfsAdmin.  The other operations createSnapshot(..), renameSnapshot(..) and deleteSnapshot(..) are actually user operations and they are declared in FileSystem.  Users can take snapshots for their own directories once admin has allowed snapshots for those directories.  Snapshot is not a HDFS-specific operation.  Many other file systems do support it.  No?

Tsz-Wo
________________________________
 From: Aaron T. Myers <[EMAIL PROTECTED]>
To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>
Sent: Wednesday, April 17, 2013 6:45 PM
Subject: Re: Heads up - Snapshots feature merge into trunk
 

I'm very excited to see that this project is nearing completion. I've been
following the development pretty closely and am very much looking forward
to getting this merged to trunk.

One thing that I do think we should address before the merge is moving the
programmatic APIs for working with snapshots. I've brought this up before,
and was told that it would be done in a separate JIRA, but I don't think
that JIRA was ever filed.

As it stands right now, the API for using snapshots is the following:

1. The API to create/delete/rename snapshots are in FileSystem.
2. The API to mark directories as snapshottable or not only exists in
DistributedFileSystem and DFSAdmin, neither of which are intended to be
public APIs.

In my opinion (and I think this was shared by others at the last snapshots
design meetup?) we should move #1 out of the FileSystem class since these
are primarily administrative APIs, and it is unlikely that any other
FileSystem implementation besides HDFS will ever implement these commands.
Also, #2 should really be in some public (not necessarily stable, but
public) class for use by tools which are used to administer HDFS. In my
opinion the most natural place for both of these APIs is in the HdfsAdmin
class, which is a public/evolving interface explicitly for these sorts of
operations.

What are others thoughts on this subject?

Best,
Aaron

--
Aaron T. Myers
Software Engineer, Cloudera
On Sat, Apr 13, 2013 at 10:05 AM, Suresh Srinivas <[EMAIL PROTECTED]>wrote:

> Support for snapshots feature is being worked on in the jira
> https://issues.apache.org/jira/browse/HDFS-2802. This is an important and
> a
> large feature in HDFS. Please see a brief presentation that describes the
> feature at a highlevel from the Snapshot discussion meetup we had a while
> back -
> https://issues.apache.org/jira/secure/attachment/12552861/Snapshots.pdf.
>
> I am exicted to announce that the feature development will soon be
> completed. Please see the jira for the design and the details of the
> subtasks. This is a heads up about the merge vote mail that will soon be
> sent.
>
> Details of development and testing:
> Development has been done in a separate branch -
> https://svn.apache.org/repos/asf/hadoop/common/branches/HDFS-2802. The
> design is posted at -
>
> https://issues.apache.org/jira/secure/attachment/12551474/Snapshots20121030.pdf
> .
> The feature development has involved close to 100 subtasks and close to 20K
> lines of code.
>
> A lot of unit tests have been added as a part of the feature. We also have
> been testing this in a cluster of 5 nodes with a long running test that
> mimics a real cluster usage with emphasis on use cases related to
> snapshots.  Please see the test plan
>
> https://issues.apache.org/jira/secure/attachment/12575442/snapshot-testplan.pdffor
> the details.
>
> Next steps, before calling for merge vote, we need to get the following
> done:
> - Add user documentation that describes the feature, and how to use it
> - Complete some of the pending tasks
> - Continue testing the feature and fix any bugs that might come up
> - Update the design document
>
> Thanks to everyone who has participated in design and development of this
> feature. Please review the work and help in testing the feature.