PCI Passthrough with both GPU and USB

tzerod

I have a fully updated xcp 8.2 running with 2 nvidia quadro rtx 6000s.

GPU passthrough works as expected.
GPU PCI passthrough works as expected.
USB PCI passthrough of an ASMedia controller works as expected

However I cannot combine GPU and USB pci passthrough. Starting the vm just hangs. I see no errors, nothing of note in xl dmesg.

I have done this before with KVM on Fedora linux, on the same machine. So I know its not the hardware or bios.

Is there some underlying reason why USB and GPU PCI Passthrough together will not work?

olivierlambert

Weird, anyone else experiencing the same issue?

mittens

I was going to create a new post dealing with this issue. So I'm going to bump this one. I am having the same issue. USB and GPU passthrough works great but not at the same time. I have not looked at the logs yet other than the quick error at the bottom of xcp-ng center

xcp-ng 8.2
center 20.04.01
hardware hp z620 with updated bios to the latest

guests:
windows 10
windows 11
ubuntu 20.04

all have the same error

olivierlambert

Are you sure you are fully up to date?

mittens

Thank you, the builds were fairly recent but the update to the hosts worked. USB and GPU passthrough are currently working together now. It did break my windows 11 vm due to no tpm/secure boot but it was a reghack to get it to work anyways

olivierlambert

Yes we fixed something recently. Always stay up to date

abufrejoval

I have used an Nvidia GTX1080ti for pass-through testing, which also combines the GPU and a USB-C controller. It works just fine for CUDA stuff, (remote) gaming via Steam on Windows isn't great but proves pass-through is working.

The only issue I had was that I had to make sure that I was putting both devices in a single line for the boot flags, the Dom0 delete and the DomU add (xe vm-param-set), and not add them line after line, as evidently these statements are not cumulative.... I wound up with only the USB controller being visible inside the VM until I re-read the documentation.

I also had nothing connected the USB controller at the time, which might easily be an issue otherwise.

Did you check that the pass-through devices were truly gone from Dom0 before you tried adding them to the VM?

Then it's evidently 2 GPUs with a USB-C controller each: were all 4 devices passed through, or was it perhaps a mix across the boards?

nvs

Hi,
I stumbled across exactly the same issue that GPU and USB pcie cards would crash the VM if passed through together. It was already mentioned in an earlier reply that updating would fix the issue, and I just wanted to confirm that works for me as well. After running the following commands on my xcp-ng 8.2:

yum update
yum upgrade

everything works nicely now! Thanks!

olivierlambert

Thanks for the feedback @nvs Glad to see we fixed those issues!