@ronan-a Any more information about RawISCSISR usage?
I can't seem to find any good match on the web.

Re: how to create a VM and with an imported VHD disk?

Looking at this thread to understand how to create a new VM without creating a Disk, I followed @Danp advice to create the VM without disk to Attach one later on.

Unfortunately, there doesn't seem to be a way to attach an entire iSCSI SR as a Disk to a VM.

To provide context about what I'm trying to do:
I have an UEFI VM running under TrueNAS where the main storage is a Zvol. Trying to transfer the VM to XCP-NG, I can share this zvol with iSCSI and mount the zvol on XCP-NG with XOA.

But AFAIK, there aren't any option to directly use the iSCSI storage as a VM disk.
(On Proxmox there's this option called "Use LUN directly" which does exactly this)

Here's what is on the zvol:

Disk /dev/zvol/tank/ubuntudockerhost-0vhbns-clone: 80 GiB, 85899345920 bytes, 167772160 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 16384 bytes
I/O size (minimum/optimal): 16384 bytes / 33554432 bytes
Disklabel type: gpt
Disk identifier: F02887A7-E662-44BC-83A5-6905B15FF0EE

Device Start End Sectors Size Type
/dev/zvol/tank/ubuntudockerhost-0vhbns-clone1 2048 1050623 1048576 512M EFI System
/dev/zvol/tank/ubuntudockerhost-0vhbns-clone2 1050624 3147775 2097152 1G Linux filesystem
/dev/zvol/tank/ubuntudockerhost-0vhbns-clone3 3147776 167772126 164624351 78.5G Linux filesystem

And here's what I can see on XCP-NG.
I'm guessing the LVM VG created on the device means that the original partitions on the volume have been overrided.

[14:20 xcp-ng-truenas ~]# lsblk
NAME                                                                                              MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sdb                                                                                                 8:16   0    80G  0 disk 
└─VG_XenStorage--630141c0--1a17--7252--d2a0--aa31476eccbf-MGT                                     253:1    0     4M  0 lvm

Is there anyway to achieve an iSCSI mount without automatic LVM VG creation?

Could someone elaborate on the procedure to have all VMs on a host shutdown properly upon XCP-NG host shutdown please?

I tried from the host prompt:

xe host-disable
xe host-shutdown

and from XOA Host: shutdown, with warning (This will shutdown your host without evacuating its VMs. Do you want to continue?)

and rightly so the host has seemingly become unavailable (ping to its IP stops)
But then what happens is very odd:

• first the VM on it still pings for a couple minutes (yes after the host stops to answers the ping)
• then the VM stops pinging
• but AFAICS XCP-NG is not OFF

Awkwardly, I just access to the IDRAC8 entreprise license on which XCP-Ng is running, and can't SEE the proper status of XCP-NG from it.
AFAIK it's not pinging but it doesn't seem OFF either. At least the IDRAC shows it ON, and upon power cycling and reconnecting to the VM the logs shows it hasn't been cleanly shutdown.

NB: the VM has xen-guest-agent running within a container, but from what I gathered, the agent in Linux guests has no role in VM shutdown: See https://xcp-ng.org/forum/topic/10631/understanding-xe-guest-utilities/16

Also, I doubled check Proxmox: it does clean shutdown VMs, either with a "shutdown -h now" command or when triggered from GUI. And that's with a VM that has Promox guest installed.

In any case, it would be nice to have XCP-NG/XOA be able to do the same.

After a successful single VM Delta backup on NFS, with Snapshot mode Normal, changing to In Memory produces the following error:

The VM has a PCI attached device.

 TrueNAS_SCALE (xcp-ng-truenas)

    Snapshot
    Start: 2025-03-20 09:46
    End: 2025-03-20 09:46
    Error: VM_HAS_PCI_ATTACHED(OpaqueRef:15cf0bbe-a635-e870-ffa5-37fd40c97b10)This is a XenServer/XCP-ng error

Start: 2025-03-20 09:46
End: 2025-03-20 09:46
Duration: a few seconds
Error: VM_HAS_PCI_ATTACHED(OpaqueRef:15cf0bbe-a635-e870-ffa5-37fd40c97b10)This is a XenServer/XCP-ng error

If thuis is unsupported maybe it should be documented.

BTW, I can't understand how the Offline mode works fro the doc: does it actually shutdown the VM before the snapshot?

I confirm local configuration is not backup with Pool metadata backup.

AFAIK XO "Pool Metadata backup" backups the "configuration" that's been applied to an XCP-NG instance.

Would that include commands to manage PCI devices (xl pci-assignable-add) that are run locally on XCP-NG? (AFAIK it needs to be done with CLI locally, not from XO)

Anything I configure on the XCP-NG system, like say installing a Dell Service Module, or any binary installed with yum and it's configuration, how do you backup that?

I'm preparing something like a conf as code to apply to configure a fresh instance from scratch, but I'd feel better if I knew I could lose my boot device and restore from a cold storage as-is also.

I understand, thanks for sharing the plan.

I'm not sure the community is looking for something like that, but for what it's worth, here is what Proxmox is proposing for ZFS capable SAN appliances: https://pve.proxmox.com/wiki/Storage:_ZFS_over_ISCSI

AFAIU, it's a storage class based on a framework using ssh to create ZVOL and share them over iSCSI on a SAN appliance. Storage snapshots are then triggered on the SAN.
And it's apparently possible to provide your own SAN implementation as a plugin.

Sounds like part of a solution to me, but I have no idea whether it would be suitable/desirable for XCP-NG.

thanks, I found out Settings->Plugins->web-hooks which allows just this upon metadataBackup events.

Aren't there any backup integration existing for backup with SR on SAN?
Aren't typically the snapshots done on the SAN side in these cases but need to be driven by XOA ?

I thought it would be a fairly common use case for enterprise setup.

My personal use case though, is to deploy VMs on an XCP-NG instance deployed on a dedicated baremetal server and use SR from SAN/NAS storage (iSCSI & NFS/SMB) implemented by a dedicated TrueNAS server instance.

I'm fresh new XCP-NG user, currently evaluating virtualized on Proxmox with the intent to replace it as the baremetal hypervisor for VM and plug it on a TrueNAS instance for SAN/NAS SRs. So far it's a pretty exiting interface and tech stack that I'm discovering, and I like the extensive Hardware Compatibility list issued from xen server.

It seems a bit hard to automate immutable OS deployment (like CoreOS/Flatcar) because of the ignition file parameter but I'm reasonably confident I'll be able to make do somehow (an adapted PXE config or maybe https://matchbox.psdn.io which seems promising to achieve just this)

But what I still can't wrap my head around is how I will be a lot to put a backup/snapshot strategy with a ZFS based SAN/NAS.
As far as I've read, Metadata Backup seems to be what I need, but I haven't found out about any configuration/hook/plugin that would allow me to sync a metadata backup with basically a ZFS zvol/dataset snapshot.

What would you advise on a way to achieve this ?