-Re: Full stack rolling restart
Aleksandr Shulman 2014-05-29, 17:58
Yes, a full-stack rolling upgrade is possible. To perform a full-stack
rolling restart of the CDH platform, we added and tested that functionality
through Cloudera Manager, starting in CM4, running CDH4 and onward.
For HBase rolling upgrades, the only Cloudera-supported path is through
Cloudera Manager (though we've tested it without CM as well). For
HDFS/MR/YARN/ZK, it's also supported using only CDH, though you can also
use CM to do it.
For the full stack rolling upgrade including HBase, here is how Cloudera
Manager curates the process at a high level:
1. Restart all master nodes by restarting services in reverse dependency
-- Master services = HMaster, NN, ZK, JT, etc.
-- Reverse-dependency order: For example, HBase, then HDFS, then ZK (since
HDFS depends on ZK and hbase depends on HDFS).
This gets a bit more complicated if there is High-Availability enabled.
Also as a general rule, backup master services (e.g. backup master) are
upgraded before the active master services.
2. Restart all worker nodes (nodes that run worker services) in batches
(default is 1, but is configurable)
-- Worker service = DN, RS, TT, etc.
-- Reverse-dependency order: Turn off balancer. Decommission RS (by
closing and moving off all the regions one by one), gracefully shut down
DN, start the DN back up, start the RS back up, load that RS with regions.
Repeat for each worker node.
Once the master and worker services have been restarted on all nodes of the
cluster, the hbase balancer is then turned back on and the cluster is
Caveat: Rolling upgrades are only supported between minor versions of CDH.
So 4.x to 4.y OR 5.x to 5.y (but not 4.x to 5.y).
Did that answer your question?
On Thu, May 29, 2014 at 9:11 AM, Jeremy Carroll <[EMAIL PROTECTED]> wrote: