GPU AMD RX6700XT work on Windows 11 host but not on Ubuntu 22.04/24.04 and Debian 12

papaeng89

Hello,

I use vCPU passthough for AMD RX6700XT on XCP-NG 8.3 with VMs:

Windows 11 : work fine and when i restart, shutdown VM, i see always vCPU AMD RX6700XT
Ubuntu 22.04/24.04 and Debian 12 : work for the first time but when i restart or shutdown VM, the vCPU not up to VM because i have when reboot/shutdown a message on XCP-NG host

[ 1574.970142] pciback 0000:03:00.1: timed out waiting for pending transaction; performing function level reset anyway
[ 1576.218148] pciback 0000:03:00.1: not ready 1023ms after FLR; waiting
[ 1577.274087] pciback 0000:03:00.1: not ready 2047ms after FLR; waiting
[ 1579.578034] pciback 0000:03:00.1: not ready 4095ms after FLR; waiting
[ 1583.929931] pciback 0000:03:00.1: not ready 8191ms after FLR; waiting
[ 1592.377698] pciback 0000:03:00.1: not ready 16383ms after FLR; waiting
[ 1610.809303] pciback 0000:03:00.1: not ready 32767ms after FLR; waiting
[ 1645.624486] pciback 0000:03:00.1: not ready 65535ms after FLR; giving up

when VM restart the vCPU not see in VM and i see same message on XCP-NG host. You need to restart the host to see the vCPU for Linux VM machines.

when i launch lspci on host , the graphic card on pci is :

03:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Navi 22 [Radeon RX 6700/6700 XT/6750 XT / 6800M/6850M XT] (rev ff)
03:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Navi 21/23 HDMI/DP Audio Controller (rev ff)

when i disable with udev rules , the pci devices : 03:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Navi 21/23 HDMI/DP Audio Controller. i never see message pciback on host XCP-NG server but in VM ( Windows and Linux ) the vCPU not recognized to Windows and Linux VMs. So, i need this pci device for vCPU work on VM Machines.

I don't know what to do anymore

olivierlambert

Hmm looks like a driver init issue, I have a vague recollection of a bug in the Linux driver for this, but I'm not 100% sure. Any opinion @Teddy-Astie ?

papaeng89

@olivierlambert

Finally, it's work.
you need to:

Enable Passthrough on the GPU and the GPU audio controller
Do not use GPU Passthrough for VM
Attach the 2 PCIs in the VM: 1 for the GPU and the other for the graphics card audio controller (hdmi)

This then allows you to have the right pciback and especially the link between the GPU and its audio controller.

 3.995379] pciback 0000:03:00.0: seizing device
[    3.995385] pciback 0000:03:00.1: seizing device
[    3.997912] pciback 0000:03:00.0: Video device with shadowed ROM at [mem 0x000c0000-0x000dffff]
[    3.997918] pciback 0000:03:00.1: Linked as a consumer to 0000:03:00.0

As the VM will be used to run Ollama

I also need the AVX functions of the CPU and for that I found an article on the forum where we are given the information to activate it in the VM if it is not available by default ( https://xcp-ng.org/forum/topic/6836/avx/14 ) with an old more functional link at Citrix.
You need to use the function: platform:featureset
The following example enables AVX:

xe vm-param-set uuid=$VM_UUID platform:featureset=1fcbfbff:f7fa3223:ee500800:042109f7:0000000f:fdbf4fab:02405f4f:00000000:750fd005:9c00040c:00000020:00000000:00000000:00000001:00000000:00000000:00000000:00000000

olivierlambert

Great news! I wonder why AVX isn't directly passed to the VM though.

papaeng89

@olivierlambert Because on the pool, one server haven't AVX .

I'm on another issue with extremely long loading time on Ollama models compared to testing on the same machine without xcp-ng

olivierlambert

Ah indeed it makes sense then

papaeng89

This post is deleted!