botched pool patching and now we can't change pool master
-
One of our engineers who should know better patched a host in our 8.3.0 pool before patching the master and then tried to promote it. Later they emptied, patched and rebooted the current master. I'm not sure exactly what else they did in their flailing before I stepped in a few hours ago.
Right now the current master is patched but I'm unable to change the master, I get an error "Cannot restore on this host because it was saved on an incompatible version." I tried restarting the toolstack on every host in the pool and the next attempt returned a different error "Cannot forward messages because the host cannot be contacted. The host may be switched off or there may be network connectivity problems" but that may be just because the dust hadn't settled from the toolstack restarts yet? I waited ten minutes and tried again and got the first error again.
I could continue emptying, patching, rebooting the rest of the hosts but I don't want to leave this pool in an unknown state and find out later on things are broken under the hood.
I'd be grateful for any guidance; this is our production pool and if I'm not confident it's healthy we'll start the painful process of creating a new pool and slow cross-pool VM migrations.
Update: I also can't create new VMs.
-
@randyrue Was the current master rebooted after being patched?
-
@Danp rebooted it again to be sure. No change, I still get the first error
-
It isn't clear why you are trying to change the master at this time. Can you explain? What is the status of the two hosts that have been patched and rebooted?
