I'd like to propose a potential solution to testing HBase Rolling Upgrade
between minor versions of HBase. It would be great to get feedback.
This is a tricky feature and it would be really strengthen our trust with
the product users if we can show that it is stable and correct (e.g. no
loss of data). The best way to do that is with a test framework that allows
for repeatability, transparency, and reporting.
It would be great to see us all agree on what should be supported during a
rolling upgrade and work towards it using this test framework.
To that end, I have a few goals in mind:
-Integration test should be able to trigger a rolling upgrade of an
existing cluster, regardless of how it is set up
-It should be extensible enough so that it can be extended to support
different ways of setting up HBase (tarballs, packages, parcels, or any
other formats that are relevant)
-It should be able to report results of various testing operations
-It should be easy to add new tests to the suite, particularly by those who
have only passing familiarity with rolling upgrade
-These tests should be first-class citizens WRT other tests
-Should standardize what the minimum set of services is for a rolling
upgrade (e.g. master, backup master, 2 regionservers, zk, etc.)
-Should support both secure and insecure rolling upgrade
I propose using IntegrationTest to bind to an existing remote cluster. It
would be able to perform the rolling upgrade steps while running tests over
a cluster that is already in place.
At a high level, we would first agree on and create a standard workflow for
upgrading a cluster. Then, we would work on using IntegrationTest to run
tests (e.g. insert data, do mapreduce over hbase jobs, etc.) while the
cluster is in flux. The challenge would be adding code that will handle
specific cluster setups (e.g. tarballs vs. packages vs. parcels), but we
can abstract it away.
Please let me know what you think about this idea and if you'd like to work
together to accomplish this goal.