GPU passthrough - Windows VM doesn't boot correctly

Morpheus0x

OS: XCP-ng 8.1.0
Guest OS: Windows 10 LTSC 2019, Version 1809, Build 17763.2183
GPU: Nvidia GTX 1660 Super

The GPU worked for the XCP-ng console before I followed the PCIe Passthough guide from the XCP-ng docs. I enabled RDP and installed Parsec on the Windows VM. XCP-ng Guest Tools are installed. I am unable to install the Nvidia drivers beforehand. I get following error: This NVIDIA graphics driver is no compatible with this version of Windows. This graphics driver could not find compatible graphics hardware. I tried 465.89 and 471.96, both show the same error. I think this is because the GPU isn't passed through yet. The VM should boot even without those drivers.
Following PCIe devices:

[23:38 xcp-ng ~]# lspci

09:00.0 VGA compatible controller: NVIDIA Corporation Device 21c4 (rev a1)
09:00.1 Audio device: NVIDIA Corporation Device 1aeb (rev a1)
09:00.2 USB controller: NVIDIA Corporation Device 1aec (rev a1)
09:00.3 Serial bus controller [0c80]: NVIDIA Corporation Device 1aed (rev a1)

[23:42 xcp-ng ~]# xl pci-assignable-list 
0000:09:00.0
0000:09:00.3
0000:09:00.1
0000:09:00.2

I assigned all four device to my Windows VM:

[23:45 xcp-ng ~]# xe vm-param-set other-config:pci=0/0000:09:00.0,0/0000:09:00.1,0/0000:09:00.2,0/0000:09:00.3 uuid=<uuid>

After that the devices still show up under xl pci-assignable-list, is that expected?
Starting the Windows VM works without an error.

[23:48 xcp-ng ~]# xe vm-start uuid=<uuid>
[23:48 xcp-ng ~]#

There is no video out on the console, which is expected I think, but the Windows VM never boots correctly. I am unable to ping/establish rdp connection. I have also tried connecting a monitor to the GPU while the VM is running and before booting, but get no video out.

This is the only GPU installed on the system, no integrated graphics(Ryzen 5 3600). This shouldn't be a problem right? XCP-ng boots normally, the screen goes blank in the middle of the boot process, indicating that it's correctly blacklisted.

Is this a known problem? How do I fix this?

UPDATE:
I got the VM to boot with the gpu passed through by NOT passing throught the USB Controller(0000:09:00.2). I am now able to install the nvidia drivers. Maybe this has to do with the USB Passthrough problems described here: https://xcp-ng.org/forum/topic/3594/pci-passthrough-usb-passthrough-does-not-work-together/ or maybe because my gpu doesn't have a USB-C connector which makes the USB controller irrelevant.

After restarting to apply the nvidia drivers I get a black screen on the console and it seems that the VM once again doesn't boot correctly. I am unable to ping or establish rdp connection. I forced a shutdown and starting the VM again I could see the Tiano Core Windows boot screen, before I once again get a black screen.

UPDATE 2:
I also tried TightVNC but have the same problem that the VM isn't accessible via the network.

UPDATE 3:
I was able to boot the VM with the GPU. The 1660 Super was correctly shown in the Device Manager. After opening Geforce experience and trying to download the latest update I got the black screen again. During testing I also got a BSOD with error code VIDEO_TDR_FAILURE a couple of times. It seems that every time I try to update the drivers, the VM crashes.
I also have a Grid K2 which I am going to try next.

l1c

I haven't successfully passed-through a graphics card with XCP-ng. The closest I have gotten was to pass-through the card and install the drivers, but it would not display anything. This was done with a previous version of XCP-ng, so that might be an avenue to take. I wasn't even able to install the drivers in the latest XCP-ng version.

I have, however, have had success with Qubes OS, so this might not apply 1 to 1.

Basically there is an issue with Xen which does not allow VMs with a graphics card passed-through to boot correctly when they are assigned more than 3.5GBs of ram. There is a work around for Qubes, which gets passed this.

https://neowutran.ovh/qubes/articles/gaming_windows_hvm.html

You have to make sure that the graphics card is blacklisted at GRUB, so you shouldn't be able to see the XCP-ng console if you do it correctly, as you said.

Another problem that I've run into was installing the drivers. Basically, as soon as you are able after you've installed Windows, you should download your drivers, turn off the vm, and remove the network connection to the vm. Then, reboot the vm and install the drivers. Having the vm connected to the internet, for some reason would not allow the drivers to install correctly. The graphics card worked correctly after that.

I did run into a problem similar to yours when I passed-through my motherboard's usb's, however. With a 5600xt everything worked perfectly, but with a gtx 1070, the vm became unusable after passing-though the usb's, so this might be an issue with nvidia.

Since the issue is with Xen you might be able to duplicate the process above in XCP-ng.

Keep us posted if it works.

Morpheus0x

@l1c Thanks for the reply and the Qubes OS workaround . I didn't now that Qubes was also using Xen, that's awesome. I will try with less ram (Currently 8GB), hopefully it will work. And if it does I will look into the workaround.

I found this issue on Github:
https://github.com/xcp-ng/xcp/issues/435
Could this be the problem I am having?

I am not at home this weekend and will give an update next week.

olivierlambert created this issue in xcp-ng/xcp

open q35 support in Xen #435

l1c

@toby4213
I don't think since it seems from the link you posted that the latest Xen version resolve the issue.
TBH most of this goes way above my head...

tony

@toby4213 said in GPU passthrough - Windows VM doesn't boot correctly:

This is the only GPU installed on the system, no integrated graphics(Ryzen 5 3600). This shouldn't be a problem right?

I had this problem before and XCP-ng won't boot without another graphic card to display its xsconsole. It might have been changed in latest version of XCP-ng but if not you'll have to have at least 1 graphics card for the host (can even be an iGPU) and another PCIe card for the VM.

Also support for GeForce graphics card for VM is still in BETA stage so it might not work 100%, if you want to be sure then try your GRID card which might work better. AMD cards will work for sure even with vGPU.

Morpheus0x

@l1c said in GPU passthrough - Windows VM doesn't boot correctly:

@toby4213
I don't think since it seems from the link you posted that the latest Xen version resolve the issue.
TBH most of this goes way above my head...

Where do you see this as implemented? The github issue is still open...

@tony said in GPU passthrough - Windows VM doesn't boot correctly:

I had this problem before and XCP-ng won't boot without another graphic card to display its xsconsole. It might have been changed in latest version of XCP-ng but if not you'll have to have at least 1 graphics card for the host (can even be an iGPU) and another PCIe card for the VM.

Also support for GeForce graphics card for VM is still in BETA stage so it might not work 100%, if you want to be sure then try your GRID card which might work better. AMD cards will work for sure even with vGPU.

XCP-ng does boot with the GPU passed thorugh. XCP-ng comes online in XOA.
The problem is the unstable Windows VM.
I ordered a PCI riser, since I don't have a free 16x slots to test with 2 GPUs installed and 1 of them passed through.

l1c

@toby4213
Oh I see. I may have read incorrectly. I honestly would not be able to tell you if that is the reason why you are having issues. I do know, however, that the 3.5GB of ram error is fairly common for people using Xen and wanting to pass-though a graphics card. If what you send is linked to that error, then it might be related.

Greg_E

The only GPU I've tried to pass through has been an old Quadro 2000 card, and it seemed to work fine (though the old driver is a bit difficult on Win Server 2022). But the host also has a cheapo server GPU for XCP-NG to use for boot.

I mention Quadro because those didn't have the crippled firmware blocking them from pass through like the Geforce cards have/had. Kind of a dirty trick from nVidia!

Sorry I don't have any newer cards that I can test with, and I also only have Quadro cards here at all, "required" for some of our software.

What I did find out is that if you pass through a GPU, you can not migrate that VM to another host, even if that other host has the same model GPU installed. And with that my GPU pass though testing is complete.