We had to re-IP our entire cluster recently to change subnetworks, and we
essentially lost everything (it was development, so no big deal). However,
doing a re-IP operation may be required in actual operational cases, and
I'd like to know if it can be done or not so we can note it for the future
(as in document "what not to do" to avoid data loss).
The issue we had was that after shutting down the cluster, re-IPing all
servers, and starting everything back up, the tablets were still assigned
to Tabletservers with the old IP addresses, even though all the hostnames
were the same. So the system showed 3 Tabletservers, but no tablets, and
no entries in the tables where previously there were 400 million.
A) Does Zookeeper track Tabletservers by IP address only, and not hostname?
B) If A is true, is there a mechanism to change those entries in Zookeeper
so that a re-IP operation could be performed?