Resolved. Patched the rest of the hosts. Restarted all the toolstacks for good measure. I can move the master role now, and deploy new VMs.

@Danp To answer your earlier question about the state of the patched hosts, three hosts are currently fully patched with one of them as the current pool master. I've tried to promote both of the other two with the same "Cannot restore" error.

For the sake of completeness I just tried promoting one of the non-patched hosts. Same error.

And I'm still unable to deploy a new VM ("NOT_SUPPORTED_DURING_UPGRADE").

@Danp I believe so. I was not driving at the time.

@Danp I was attempting to change the host as we'll be retiring the current master. In any case if I can't promote another host this would suggest our production pool is otherwise wedged. The same for not being able to create a new VM.

@Danp rebooted it again to be sure. No change, I still get the first error

One of our engineers who should know better patched a host in our 8.3.0 pool before patching the master and then tried to promote it. Later they emptied, patched and rebooted the current master. I'm not sure exactly what else they did in their flailing before I stepped in a few hours ago.

Right now the current master is patched but I'm unable to change the master, I get an error "Cannot restore on this host because it was saved on an incompatible version." I tried restarting the toolstack on every host in the pool and the next attempt returned a different error "Cannot forward messages because the host cannot be contacted. The host may be switched off or there may be network connectivity problems" but that may be just because the dust hadn't settled from the toolstack restarts yet? I waited ten minutes and tried again and got the first error again.

I could continue emptying, patching, rebooting the rest of the hosts but I don't want to leave this pool in an unknown state and find out later on things are broken under the hood.

I'd be grateful for any guidance; this is our production pool and if I'm not confident it's healthy we'll start the painful process of creating a new pool and slow cross-pool VM migrations.

Update: I also can't create new VMs.

After reading up on xl you're not wrong. I'll fix the script and firmly chastise the guy that wrote it (in the mirror). I think I went with xl because it was easier to split the results in python.

In this case I'm only using it to return VM info for an export script. The script takes the VM name as an argument and then walks the hosts and vm lists looking for it. But the VM was cloned and has a "name" of <name-label><snapshot_stamp><_COPY>

I'm trying to change the "name" attribute of a VM that's returned by the "xl vm-list" command. This is not the same as the "name-label" attribute returned by "xe vm-list" and shown in XOA. No sign of any such attribute in the auto completion for the xe vm-param-set command.

I could clone the VM to the correct name but that feels like overkill.

@randyrue Confirmed that after cold booting a VM to a new host I can then migrate it live to another new host.