Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # dev >> Let's discuss Snapshots Feature Testing


Copy link to this message
-
Re: Let's discuss Snapshots Feature Testing
Yes, I am planning on filing a JIRA for that shortly.

-Aleks S.

On Mon, Jan 14, 2013 at 11:01 AM, Ted Yu <[EMAIL PROTECTED]> wrote:

> Thanks for the write up.
>
> Would the new tests be sub-tasks of HBASE-7290 ?
>
> Cheers
>
> On Mon, Jan 14, 2013 at 10:32 AM, Aleksandr Shulman <[EMAIL PROTECTED]
> >wrote:
>
> > Hi everyone,
> >
> > I'd like to start a thread about Cloudera's testing efforts on the
> upcoming
> > snapshots feature. This is a new feature and it's important that we
> explain
> > our testing efforts and get the community's opinion on what we'd all like
> > to see tested. My hope is that from this discussion, we can get more
> ideas
> > about what needs to be tested and gain confidence in the testing we have
> in
> > place.
> >
> > Before I begin, I'd like to introduce myself. I'm Aleks Shulman. I'm a
> > software engineer at Cloudera, working primarily on HBase. Within HBase,
> I
> > am focusing on the quality side of things. What this means to me is an
> > conversation unto itself, but in brief, I will be writing tests and test
> > frameworks. I will also be an advocate for the user experience, with
> > particular focus on API compatibility and ease-of-use.
> >
> > So let's discuss snapshots:
> > There are two main areas that should be tested and they correspond nicely
> > into what can be done as unit tests and what is better left as Jenkins
> job
> > or some other automation, unit testing and non-unit testing. We've been
> > working on this for a bit, so there is already some progress in these
> > areas:
> >
> > Unit testing - In progress or completed:
> >
> > 1. HBase Snapshots Repeatability and Idempotency Test:
> > This test class verifies proper behavior with regards performing
> > restore/clone operations on tables that themselves were created as a
> clone
> > or restored from a snapshot. This is an interesting set of cases because
> of
> > the way snapshots work. They work by pointing to the original HFiles.
> > We can use these tests to verify correctness in the file system and test
> > closure under deletion of the original table.
> >
> > 2. HBase Snapshots HTable Descriptor Test
> > This test class verifies proper behavior with regards to changes to the
> > information about the table itself before and after snapshotting in the
> > 'before' table and the 'after' table.
> >
> > 3. HBase Snapshots HFileLink Test
> > This test class inspects the correctness of the HFileLink files. It looks
> > into their permissioning, the naming convention, and how they respond
> > events. Events may include an HFile being deleted or moved.
> >
> > 4. HBase Snapshots Table Dimensions Test
> > This test class inspects operations on tables that are empty, have only
> one
> > row, have one or two CFs, etc. Basically if there is an edge scenario in
> > what the table looks like, that may affect the way it snapshotted or
> > restored/cloned.
> >
> > 5. HBase Snapshots Independence Test
> > This test should verify that all aspects of table independence are
> > guaranteed between the original table and the restored snapshot/clone.
> > This includes things like data mutations, compactions, splits, etc. It
> also
> > includes metadata changes.
> >
> > 6.  HBase Snapshots Aborted or Failed Snapshot Cleanup
> > Verifies that no cruft is left over after an attempt to snapshot a table
> > fails or is aborted. We should be able to account for every file in the
> > file system before and after.
> >
> > 7. HBase Snapshots HFile Archive Test
> > This test task is to fill in any gaps in testing of archiving as it
> relates
> > to snapshots. The snapshots relies on the HFileArchiver/LogArchiver with
> > two new cleaners (SnapshotHFile/SnapshotLog Cleaners), so we'd need to go
> > through and find out what needs to be tested between them.
> >
> > 8. HBase Snapshots Export Test
> > This test should verify that export of a snapshot to another cluster
> works
> > properly.
> > Implemented as: mvn clean test -PlocalTests
> > -Dtest=org.apache.hadoop.hbase.snapshot.TestExportSnapshot

Best Regards,

Aleks Shulman
847.814.5804
Cloudera
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB