Issues with Windows 11 VM

zorrog

We're evaluating XCP-ng for running Windows VMs with GPU (AMD RX 6700 XT) passthrough. Using XCP-ng 8.3 beta 2 (iso that someone from Vates made in December), installed the most recent Win11 and got the Citrix drivers via Windows Update, plus installed the tools from Citrix website. So far, I observed a few issues.

GPU appears in Device Manager but doesn't work (Code 43 error).
powercfg /HIBERNATE ON reports "The system firmware does not support hibernation."
According to Nova bench, disk performance is ~60% lower than with other hypervisors on the same hardware (QEMU, VBox, ESXi, Hyper-V). While investigating this, I noticed that the disk device doesn't support "Enable write caching on the device" option, which, I suspect, could be causing the low benchmark number.

I didn't have those issues with QEMU/Proxmox on the same hardware.
Any ideas/hints much appreciated. Hardware is not set in stone, as long as there are guarantees to be working without issues and it's in roughly the same price category.

olivierlambert

Hi,

I can't tell for the GPU, are you sure you are passing through everything GPU related? (like you might have 2 devices, including the sound/HDMI one, not just the GPU itself). Those partial passthrough could create similar problems.

Regarding hibernation, you could use the suspend feature of XCP-ng, that will save the memory on a storage repository, so you can restore pretty fast with all the RAM/context unchanged later on.

Finally, benching is far more complex that one can imagine. I have no idea what are you benching exactly (the kind of storage) and be advised it's very different in Xen than KVM/Qemu for example. On XCP-ng, it tend to scale pretty well with more disks, so when you compare one disk to one disk, the result can be better on something else, while in a real production context, XCP-ng can perform better with many VMs/many disks working at once. So keep that in mind when you do a benchmark, because it's sometimes the result is not what are you thinking when you want to go in real production scenario

zorrog

@olivierlambert yes, passing both sound and VGA devices.

Does suspend work with PCI passthrough? It doesn't in other hypervisors, so I assumed it's a no-go. If it does - great, I'll give it a shot once I manage to pass through GPU.
As an alternative to the full passthrough, is there a compatibility list for vGPU compatibility? Would any AMD card work?

Re benchmarks - sure, I take it with a pinch of salt at this stage, it was more curiosity about missing write caching.

olivierlambert

I never tried myself to suspend a VM with an attached PCI device, so I can't tell

Regarding vGPU, it works withy MxGPU-enabled feature on some cards (now old AMD Firepro).

Let's try to see in the community if people had a similar issue than you with a recent AMD card in PCI passthrough

planedrop

I've got passthrough to work a number of times without issue, the only thing I had to make sure of was that all devices related to the GPU were passed through completely.

Are you following the docs step by step?

I have a Ubuntu VM running with a 2060 passed through right now, works flawlessly and even survived a power loss on the host.