GPU passthrough - Windows VM doesn't boot correctly
OS: XCP-ng 8.1.0
Guest OS: Windows 10 LTSC 2019, Version 1809, Build 17763.2183
GPU: Nvidia GTX 1660 Super
The GPU worked for the XCP-ng console before I followed the PCIe Passthough guide from the XCP-ng docs. I enabled RDP and installed Parsec on the Windows VM. XCP-ng Guest Tools are installed. I am unable to install the Nvidia drivers beforehand. I get following error:
This NVIDIA graphics driver is no compatible with this version of Windows. This graphics driver could not find compatible graphics hardware.I tried 465.89 and 471.96, both show the same error. I think this is because the GPU isn't passed through yet. The VM should boot even without those drivers.
Following PCIe devices:
[23:38 xcp-ng ~]# lspci 09:00.0 VGA compatible controller: NVIDIA Corporation Device 21c4 (rev a1) 09:00.1 Audio device: NVIDIA Corporation Device 1aeb (rev a1) 09:00.2 USB controller: NVIDIA Corporation Device 1aec (rev a1) 09:00.3 Serial bus controller [0c80]: NVIDIA Corporation Device 1aed (rev a1) [23:42 xcp-ng ~]# xl pci-assignable-list 0000:09:00.0 0000:09:00.3 0000:09:00.1 0000:09:00.2
I assigned all four device to my Windows VM:
[23:45 xcp-ng ~]# xe vm-param-set other-config:pci=0/0000:09:00.0,0/0000:09:00.1,0/0000:09:00.2,0/0000:09:00.3 uuid=<uuid>
After that the devices still show up under
xl pci-assignable-list, is that expected?
Starting the Windows VM works without an error.
[23:48 xcp-ng ~]# xe vm-start uuid=<uuid> [23:48 xcp-ng ~]#
There is no video out on the console, which is expected I think, but the Windows VM never boots correctly. I am unable to ping/establish rdp connection. I have also tried connecting a monitor to the GPU while the VM is running and before booting, but get no video out.
This is the only GPU installed on the system, no integrated graphics(Ryzen 5 3600). This shouldn't be a problem right? XCP-ng boots normally, the screen goes blank in the middle of the boot process, indicating that it's correctly blacklisted.
Is this a known problem? How do I fix this?
I got the VM to boot with the gpu passed through by NOT passing throught the USB Controller(
0000:09:00.2). I am now able to install the nvidia drivers. Maybe this has to do with the USB Passthrough problems described here: https://xcp-ng.org/forum/topic/3594/pci-passthrough-usb-passthrough-does-not-work-together/ or maybe because my gpu doesn't have a USB-C connector which makes the USB controller irrelevant.
After restarting to apply the nvidia drivers I get a black screen on the console and it seems that the VM once again doesn't boot correctly. I am unable to ping or establish rdp connection. I forced a shutdown and starting the VM again I could see the Tiano Core Windows boot screen, before I once again get a black screen.
I also tried TightVNC but have the same problem that the VM isn't accessible via the network.
I was able to boot the VM with the GPU. The 1660 Super was correctly shown in the Device Manager. After opening Geforce experience and trying to download the latest update I got the black screen again. During testing I also got a BSOD with error code VIDEO_TDR_FAILURE a couple of times. It seems that every time I try to update the drivers, the VM crashes.
I also have a Grid K2 which I am going to try next.
I haven't successfully passed-through a graphics card with XCP-ng. The closest I have gotten was to pass-through the card and install the drivers, but it would not display anything. This was done with a previous version of XCP-ng, so that might be an avenue to take. I wasn't even able to install the drivers in the latest XCP-ng version.
I have, however, have had success with Qubes OS, so this might not apply 1 to 1.
Basically there is an issue with Xen which does not allow VMs with a graphics card passed-through to boot correctly when they are assigned more than 3.5GBs of ram. There is a work around for Qubes, which gets passed this.
You have to make sure that the graphics card is blacklisted at GRUB, so you shouldn't be able to see the XCP-ng console if you do it correctly, as you said.
Another problem that I've run into was installing the drivers. Basically, as soon as you are able after you've installed Windows, you should download your drivers, turn off the vm, and remove the network connection to the vm. Then, reboot the vm and install the drivers. Having the vm connected to the internet, for some reason would not allow the drivers to install correctly. The graphics card worked correctly after that.
I did run into a problem similar to yours when I passed-through my motherboard's usb's, however. With a 5600xt everything worked perfectly, but with a gtx 1070, the vm became unusable after passing-though the usb's, so this might be an issue with nvidia.
Since the issue is with Xen you might be able to duplicate the process above in XCP-ng.
Keep us posted if it works.
@l1c Thanks for the reply and the Qubes OS workaround . I didn't now that Qubes was also using Xen, that's awesome. I will try with less ram (Currently 8GB), hopefully it will work. And if it does I will look into the workaround.
I found this issue on Github:
Could this be the problem I am having?
I am not at home this weekend and will give an update next week.
I don't think since it seems from the link you posted that the latest Xen version resolve the issue.
TBH most of this goes way above my head...
This is the only GPU installed on the system, no integrated graphics(Ryzen 5 3600). This shouldn't be a problem right?
I had this problem before and XCP-ng won't boot without another graphic card to display its xsconsole. It might have been changed in latest version of XCP-ng but if not you'll have to have at least 1 graphics card for the host (can even be an iGPU) and another PCIe card for the VM.
Also support for GeForce graphics card for VM is still in BETA stage so it might not work 100%, if you want to be sure then try your GRID card which might work better. AMD cards will work for sure even with vGPU.