Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
HBase >> mail # dev >> Let's discuss Snapshots Feature Testing


+
Aleksandr Shulman 2013-01-14, 18:32
Copy link to this message
-
Re: Let's discuss Snapshots Feature Testing
Thanks for the write up.

Would the new tests be sub-tasks of HBASE-7290 ?

Cheers

On Mon, Jan 14, 2013 at 10:32 AM, Aleksandr Shulman <[EMAIL PROTECTED]>wrote:

> Hi everyone,
>
> I'd like to start a thread about Cloudera's testing efforts on the upcoming
> snapshots feature. This is a new feature and it's important that we explain
> our testing efforts and get the community's opinion on what we'd all like
> to see tested. My hope is that from this discussion, we can get more ideas
> about what needs to be tested and gain confidence in the testing we have in
> place.
>
> Before I begin, I'd like to introduce myself. I'm Aleks Shulman. I'm a
> software engineer at Cloudera, working primarily on HBase. Within HBase, I
> am focusing on the quality side of things. What this means to me is an
> conversation unto itself, but in brief, I will be writing tests and test
> frameworks. I will also be an advocate for the user experience, with
> particular focus on API compatibility and ease-of-use.
>
> So let's discuss snapshots:
> There are two main areas that should be tested and they correspond nicely
> into what can be done as unit tests and what is better left as Jenkins job
> or some other automation, unit testing and non-unit testing. We've been
> working on this for a bit, so there is already some progress in these
> areas:
>
> Unit testing - In progress or completed:
>
> 1. HBase Snapshots Repeatability and Idempotency Test:
> This test class verifies proper behavior with regards performing
> restore/clone operations on tables that themselves were created as a clone
> or restored from a snapshot. This is an interesting set of cases because of
> the way snapshots work. They work by pointing to the original HFiles.
> We can use these tests to verify correctness in the file system and test
> closure under deletion of the original table.
>
> 2. HBase Snapshots HTable Descriptor Test
> This test class verifies proper behavior with regards to changes to the
> information about the table itself before and after snapshotting in the
> 'before' table and the 'after' table.
>
> 3. HBase Snapshots HFileLink Test
> This test class inspects the correctness of the HFileLink files. It looks
> into their permissioning, the naming convention, and how they respond
> events. Events may include an HFile being deleted or moved.
>
> 4. HBase Snapshots Table Dimensions Test
> This test class inspects operations on tables that are empty, have only one
> row, have one or two CFs, etc. Basically if there is an edge scenario in
> what the table looks like, that may affect the way it snapshotted or
> restored/cloned.
>
> 5. HBase Snapshots Independence Test
> This test should verify that all aspects of table independence are
> guaranteed between the original table and the restored snapshot/clone.
> This includes things like data mutations, compactions, splits, etc. It also
> includes metadata changes.
>
> 6.  HBase Snapshots Aborted or Failed Snapshot Cleanup
> Verifies that no cruft is left over after an attempt to snapshot a table
> fails or is aborted. We should be able to account for every file in the
> file system before and after.
>
> 7. HBase Snapshots HFile Archive Test
> This test task is to fill in any gaps in testing of archiving as it relates
> to snapshots. The snapshots relies on the HFileArchiver/LogArchiver with
> two new cleaners (SnapshotHFile/SnapshotLog Cleaners), so we'd need to go
> through and find out what needs to be tested between them.
>
> 8. HBase Snapshots Export Test
> This test should verify that export of a snapshot to another cluster works
> properly.
> Implemented as: mvn clean test -PlocalTests
> -Dtest=org.apache.hadoop.hbase.snapshot.TestExportSnapshot
> However, we need to add more test around chmod, chown and checksums
>
> 9. HBase Snapshots Concurrent Snapshots Test
> This test class will enforce proper behavior in situations where race
> conditions can occur. For example, if one process attempts to restore a
> table and another one tries to do so simultaneously, what happens? We need
+
Aleksandr Shulman 2013-01-14, 19:15
+
Andrew Purtell 2013-01-14, 23:15
+
Jonathan Hsieh 2013-01-15, 01:27
+
Andrew Purtell 2013-01-15, 02:47
+
Jonathan Hsieh 2013-01-15, 08:55
+
Nicolas Liochon 2013-01-15, 09:25
+
Jean-Marc Spaggiari 2013-01-15, 19:01
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB