Smart Reboot blocked in XO, and no Rolling Pool Update

djingo

When I would like to smart reboot my host using XO button Advanced > Smart Reboot it tells me:

Smart Reboot failed because ... VMs have their Suspend operation blocked. Would you like to force?

At first I thought this is because some VMs had enabled "Protect from accidental shutdown". So I disabled this for all VMs. However, even after a forced smart reboot, and after restarting all VMs I still do get the same message when I initiate a Smart Reboot.

What could be the root cause of this and how can I fix it?

I think is issue is what keeps me from doing a "Rolling Pool Update".

poddingue

I don't think "Protect from accidental shutdown" is what you are hitting here.
If any VM on that host has a host-bound device attached (PCI passthrough, vGPU, or an SR-IOV VIF), XAPI refuses to suspend it with VM_HAS_PCI_ATTACHED, and Smart Reboot stops there. xe host-get-vms-which-prevent-evacuation uuid=<host-uuid> should name them.
For the Rolling Pool Update half, XO 6.7.0 (released 30 July) added a shutdownPinnedVms option that shuts those VMs down and starts them again afterwards instead of aborting the run, so upgrading might get you past it.
I could easily be wrong about which of the two you are actually hitting, though, so I would start with that command and see what comes back.

AtaxyaNetwork

@djingo Hi !

Does all your VMs have guest tools ?

djingo

@AtaxyaNetwork yes, guest tools installed

djingo

@poddingue the response is "VM_REQUIRES_SR". The VMs are storing their vdisk on a local NVME. This is crucial for this hypervisor as I do not want it to depend on network storage. Can I work around this error in such a scenario, i.e. is a Rolling Pool Update possible under these circumstances?

Does the Smart Reboot require evacuating the VMs? Technically it shouldn't as in XO there will be a warning that can be manually overridden. WHy do I get the warning in case of a Smart Reboot?

poddingue

Then, my earlier suggestion about upgrading was wrong, sorry, better to say so before you spend an evening on it.

I went and read the RPU code (it's about time!): shutdownPinnedVms only handles VM_HAS_PCI_ATTACHED, VM_HAS_VGPU and VM_HAS_SRIOV_VIF, and when anything else turns up, it hands back to XAPI's CANNOT_EVACUATE_HOST on purpose.
VM_REQUIRES_SR is not in that list, so an upgrade will not move you past it.

On the Rolling Pool Update side, the docs say all VM disks have to be on shared storage (https://docs.xcp-ng.org/management/updates), and the host reboot guide defines an agile VM as one not tied to local storage or local devices (https://docs.xcp-ng.org/guides/host-reboot).
With everything on local NVMe, that reads to me like RPU is closed to you by design rather than by bug.
The route the docs give for your case is the manual one: disable the host, shut the VMs down or migrate them, then reboot.

The Smart Reboot message looks like a separate problem. In XO's smartReboot a VM counts as suspend blocked in two different cases, either when blocked_operations.suspend is set, or when XAPI itself refuses VM.assert_operation_valid for suspend. The message only describes the first, which might be why clearing the protection changed nothing. These two will show which one you are in:

xe vm-list is-control-domain=false params=name-label,blocked-operations
xe vm-list is-control-domain=false power-state=running params=name-label,allowed-operations

If suspend is missing from allowed-operations, then XAPI is refusing it rather than a flag you set. I ran both on 8.3, so I know they print something per VM.
I have not tested any of this on a two-host pool, though, so the VM_REQUIRES_SR part is my reading of the code rather than something I reproduced.