Could someone elaborate on the procedure to have all VMs on a host shutdown properly upon XCP-NG host shutdown please?

I tried from the host prompt:

xe host-disable
xe host-reboot

and from XOA Host: shutdown, with warning (This will shutdown your host without evacuating its VMs. Do you want to continue?)

and rightly so the host has seemingly become unavailable (ping to its IP stops)
But then what happens is very odd:

• first the VM on it still pings for a couple minutes (yes after the host stops to answers the ping)
• then the VM stops pinging
• but AFAICS XCP-NG is not OFF

Awkwardly, I just access to the IDRAC8 entreprise license on which XCP-Ng is running, and can't SEE the proper status of XCP-NG from it.
AFAIK it's not pinging but it doesn't seem OFF either. At least the IDRAC shows it ON, and upon power cycling and reconnecting to the VM the logs shows it hasn't been cleanly shutdown.

NB: the VM has xen-guest-agent running within a container, but from what I gathered, the agent in Linux guests has no role in VM shutdown: See https://xcp-ng.org/forum/topic/10631/understanding-xe-guest-utilities/16

Also, I doubled check Proxmox: it does clean shutdown VMs, either with a "shutdown -h now" command or when triggered from GUI. And that's with a VM that has Promox guest installed.

In any case, it would be nice to have XCP-NG/XOA be able to do the same.

After a successful single VM Delta backup on NFS, with Snapshot mode Normal, changing to In Memory produces the following error:

The VM has a PCI attached device.

 TrueNAS_SCALE (xcp-ng-truenas)

    Snapshot
    Start: 2025-03-20 09:46
    End: 2025-03-20 09:46
    Error: VM_HAS_PCI_ATTACHED(OpaqueRef:15cf0bbe-a635-e870-ffa5-37fd40c97b10)This is a XenServer/XCP-ng error

Start: 2025-03-20 09:46
End: 2025-03-20 09:46
Duration: a few seconds
Error: VM_HAS_PCI_ATTACHED(OpaqueRef:15cf0bbe-a635-e870-ffa5-37fd40c97b10)This is a XenServer/XCP-ng error

If thuis is unsupported maybe it should be documented.

BTW, I can't understand how the Offline mode works fro the doc: does it actually shutdown the VM before the snapshot?

I confirm local configuration is not backup with Pool metadata backup.

AFAIK XO "Pool Metadata backup" backups the "configuration" that's been applied to an XCP-NG instance.

Would that include commands to manage PCI devices (xl pci-assignable-add) that are run locally on XCP-NG? (AFAIK it needs to be done with CLI locally, not from XO)

Anything I configure on the XCP-NG system, like say installing a Dell Service Module, or any binary installed with yum and it's configuration, how do you backup that?

I'm preparing something like a conf as code to apply to configure a fresh instance from scratch, but I'd feel better if I knew I could lose my boot device and restore from a cold storage as-is also.

I understand, thanks for sharing the plan.

I'm not sure the community is looking for something like that, but for what it's worth, here is what Proxmox is proposing for ZFS capable SAN appliances: https://pve.proxmox.com/wiki/Storage:_ZFS_over_ISCSI

AFAIU, it's a storage class based on a framework using ssh to create ZVOL and share them over iSCSI on a SAN appliance. Storage snapshots are then triggered on the SAN.
And it's apparently possible to provide your own SAN implementation as a plugin.

Sounds like part of a solution to me, but I have no idea whether it would be suitable/desirable for XCP-NG.

thanks, I found out Settings->Plugins->web-hooks which allows just this upon metadataBackup events.

Aren't there any backup integration existing for backup with SR on SAN?
Aren't typically the snapshots done on the SAN side in these cases but need to be driven by XOA ?

I thought it would be a fairly common use case for enterprise setup.

My personal use case though, is to deploy VMs on an XCP-NG instance deployed on a dedicated baremetal server and use SR from SAN/NAS storage (iSCSI & NFS/SMB) implemented by a dedicated TrueNAS server instance.

I'm fresh new XCP-NG user, currently evaluating virtualized on Proxmox with the intent to replace it as the baremetal hypervisor for VM and plug it on a TrueNAS instance for SAN/NAS SRs. So far it's a pretty exiting interface and tech stack that I'm discovering, and I like the extensive Hardware Compatibility list issued from xen server.

It seems a bit hard to automate immutable OS deployment (like CoreOS/Flatcar) because of the ignition file parameter but I'm reasonably confident I'll be able to make do somehow (an adapted PXE config or maybe https://matchbox.psdn.io which seems promising to achieve just this)

But what I still can't wrap my head around is how I will be a lot to put a backup/snapshot strategy with a ZFS based SAN/NAS.
As far as I've read, Metadata Backup seems to be what I need, but I haven't found out about any configuration/hook/plugin that would allow me to sync a metadata backup with basically a ZFS zvol/dataset snapshot.

What would you advise on a way to achieve this ?