Have been toying with the idea of automating the process of running
the tests that constitute the release test plan (the initial tests
would be from https://docs.google.com/spreadsheet/ccc?key=0AvLqcVIqNtlTdE9iemw2STVmWHFVMVVNY2JvZ0NwZGc#gid=0,
and they would be run with/without security turned on in the cluster).
As we continue to develop more system/largescale tests, we would keep
adding them to the harness.
I have set the ball rolling on this within the company and I hope the
community will be interested in seeing such a thing happen.
The idea is to be able to easily run a release artifact through a
series of tests (on our internal cluster or on AWS), and the result
would be published in a machine on AWS (visible to all). In the
future, this could be extended to run the tests on trunk artifacts as
well (catch regressions early).