botched pool patching and now we can't change pool master
-
One of our engineers who should know better patched a host in our 8.3.0 pool before patching the master and then tried to promote it. Later they emptied, patched and rebooted the current master. I'm not sure exactly what else they did in their flailing before I stepped in a few hours ago.
Right now the current master is patched but I'm unable to change the master, I get an error "Cannot restore on this host because it was saved on an incompatible version." I tried restarting the toolstack on every host in the pool and the next attempt returned a different error "Cannot forward messages because the host cannot be contacted. The host may be switched off or there may be network connectivity problems" but that may be just because the dust hadn't settled from the toolstack restarts yet? I waited ten minutes and tried again and got the first error again.
I could continue emptying, patching, rebooting the rest of the hosts but I don't want to leave this pool in an unknown state and find out later on things are broken under the hood.
I'd be grateful for any guidance; this is our production pool and if I'm not confident it's healthy we'll start the painful process of creating a new pool and slow cross-pool VM migrations.
Update: I also can't create new VMs.
-
@randyrue Was the current master rebooted after being patched?
-
@Danp rebooted it again to be sure. No change, I still get the first error
-
It isn't clear why you are trying to change the master at this time. Can you explain? What is the status of the two hosts that have been patched and rebooted?
-
@Danp I was attempting to change the host as we'll be retiring the current master. In any case if I can't promote another host this would suggest our production pool is otherwise wedged. The same for not being able to create a new VM.
-
It sounds like you tried to switch the pool master to a host at a lower xapi level. Had the host that you designated as the new pool master already been patched?
-
@Danp I believe so. I was not driving at the time.
-
@Danp To answer your earlier question about the state of the patched hosts, three hosts are currently fully patched with one of them as the current pool master. I've tried to promote both of the other two with the same "Cannot restore" error.
For the sake of completeness I just tried promoting one of the non-patched hosts. Same error.
And I'm still unable to deploy a new VM ("NOT_SUPPORTED_DURING_UPGRADE").
