@timemaster5 @dustyarmstrong

I published a thread with packages to test, if you have a non production host on which you can test it would be awesome: https://xcp-ng.org/forum/topic/12523/igpu-pci-passthrough-new-experimental-packages-need-testers
Thanks!

Regards,

Yann

@steff22 Good news! And thanks a lot for the test! Could you share the guest VM dmesg please? as well as the lspci -n -vvv -s SBDF of the GPU?
Also, was it working with the stock xcp-ng packages?

@ovicz Thanks a lot for the feedbacks and tests

@ovicz It seems your gpu works even without vbios since in your dmesg I can see

[    7.424250] i915 0000:00:08.0: Invalid PCI ROM header signature: expecting 0xaa55, got 0x0000
[    7.424259] i915 0000:00:08.0: [drm] Failed to find VBIOS tables (VBT)

So you're saying that anyway your setup was working even without all those new packages?
Thanks for testing anyway!

@ovicz with the new edk2 package it should help your VM access the vbios in the rom bar.
Does the VM see the gpu?
Can you share the VM dmesg as attached file?
Also, are you trying to passthrough a sriov virtual function? or the entire physical gpu?
Thanks

Another possibility for you would be to yum downgrade qemu and the xapi packages, everything except edk2 and try again.

@ovicz in your case it seems the vbios is in the ROM BAR.
You can dump the vbios by doing something like

cd /sys/bus/pci/devices/0000:00:02.0/
echo 1 > rom
cat rom > /root/vbios.bin
echo 0 > rom

@ovicz a dit:

intel alder lake cpu

Thanks for giving this a try!
Can you give me the output of lspci -n -vvv -s SBDF_OF_GPU on your dom0 please?

@antest Hi

I notice in your domU dmesg:

[    2.425816] i915 0000:00:08.0: Invalid PCI ROM header signature: expecting 0xaa55, got 0x4556
[    2.425820] i915 0000:00:08.0: [drm] Failed to find VBIOS tables (VBT)

If you are on a non-production machine can you give this a try?
=> https://xcp-ng.org/forum/topic/12523/igpu-pci-passthrough-new-experimental-packages-need-testers

Thanks!
Yann

@jp13232 @peek have you tried following this: https://docs.xcp-ng.org/compute/#pci-passthrough ?

Also, you can have a look there: https://xcp-ng.org/forum/topic/12523/igpu-pci-passthrough-new-experimental-packages-need-testers if you finally manage to do the passthrough but have "vbios issues" like explained in the thread.

@the_spice @bullerwins @xerxist @hawkpro @vhaelan @ovicz

Hi

I published a post this morning with packages to test iGPU passthrough, if you have a non production host on which you can test it would be awesome: https://xcp-ng.org/forum/topic/12523/igpu-pci-passthrough-new-experimental-packages-need-testers
Thanks!

Regards,

Yann

@mgr42 Hi

I published a post this morning with packages to test, if you have a non production host on which you can test it would be awesome: https://xcp-ng.org/forum/topic/12523/igpu-pci-passthrough-new-experimental-packages-need-testers
Thanks!

Regards,

Yann

DO NOT TEST THIS IN PRODUCTION

I did a Proof-of-Concept to improve the iGPU passthrough situation in the case where the vbios is not in the ROM BAR. This would usually fail in the guest because the guest amdgpu or i915driver would not be able to find the vbios.
I would like to share this PoC with you to gather feedback / testing.

This is not at all a production ready solution, as you will see with the manual steps which are needed to test this.

Who should test this?

If you have an host with an iGPU, and the usual 8.3 PCI passthrough procedure described at https://docs.xcp-ng.org/compute/#pci-passthrough does not work because the guest dmesg prints something like

Invalid PCI ROM header signature: expecting 0xaa55, got 0x4556

or

Failed to find VBIOS tables (VBT)

or

Unable to locate a BIOS ROM

What has been tested so far

I only tested on an AMD Ryzen Embedded V2748 with Radeon Graphics embedded APU system with a Radeon RX Vega series integrated GPU (PCI vendor id 1002 PCI device id 1636). With this config, I tested that the guest VM was able to correctly probe the GPU with the amgdpu driver.

How to test

• Install a brand new XCP-ng 8.3 on a test host (or make sure you are up to date with latest 8.3 packages on your test host)
• Configure XCP-ng user repositories:

wget https://koji.xcp-ng.org/repos/user/8/8.3/xcpng-users.repo -O /etc/yum.repos.d/xcpng-users.repo

• Update your system with the iGPU PoC packages

yum update --enablerepo=xcp-ng-ysionneau1

• it will prompt you about importing a new GPG key
• it should install the following packages:

=============================================================================================================================================================================================================================================================================================================================
 Package                                                                      Arch                                                            Version                                                                                       Repository                                                                  Size
=============================================================================================================================================================================================================================================================================================================================
Updating:
 edk2                                                                         x86_64                                                          20220801-1.7.11.3~romtest.2.xcpng8.3                                                          xcp-ng-ysionneau1                                                          3.0 M
 forkexecd                                                                    x86_64                                                          26.1.19-1.2.0.romtest.1.xcpng8.3                                                              xcp-ng-ysionneau1                                                          2.5 M
 message-switch                                                               x86_64                                                          26.1.19-1.2.0.romtest.1.xcpng8.3                                                              xcp-ng-ysionneau1                                                          4.6 M
 qcow-stream-tool                                                             x86_64                                                          26.1.19-1.2.0.romtest.1.xcpng8.3                                                              xcp-ng-ysionneau1                                                          1.4 M
 qemu                                                                         x86_64                                                          2:4.2.1-5.2.18.1.0.fixrom.1.xcpng8.3                                                          xcp-ng-ysionneau1                                                           16 M
 rrdd-plugins                                                                 x86_64                                                          26.1.19-1.2.0.romtest.1.xcpng8.3                                                              xcp-ng-ysionneau1                                                          9.9 M
 sm-cli                                                                       x86_64                                                          26.1.19-1.2.0.romtest.1.xcpng8.3                                                              xcp-ng-ysionneau1                                                          4.9 M
 squeezed                                                                     x86_64                                                          26.1.19-1.2.0.romtest.1.xcpng8.3                                                              xcp-ng-ysionneau1                                                          1.9 M
 varstored-guard                                                              x86_64                                                          26.1.19-1.2.0.romtest.1.xcpng8.3                                                              xcp-ng-ysionneau1                                                          5.0 M
 vhd-tool                                                                     x86_64                                                          26.1.19-1.2.0.romtest.1.xcpng8.3                                                              xcp-ng-ysionneau1                                                          5.0 M
 wsproxy                                                                      x86_64                                                          26.1.19-1.2.0.romtest.1.xcpng8.3                                                              xcp-ng-ysionneau1                                                          1.1 M
 xapi-core                                                                    x86_64                                                          26.1.19-1.2.0.romtest.1.xcpng8.3                                                              xcp-ng-ysionneau1                                                           30 M
 xapi-nbd                                                                     x86_64                                                          26.1.19-1.2.0.romtest.1.xcpng8.3                                                              xcp-ng-ysionneau1                                                          3.1 M
 xapi-rrd2csv                                                                 x86_64                                                          26.1.19-1.2.0.romtest.1.xcpng8.3                                                              xcp-ng-ysionneau1                                                          3.0 M
 xapi-storage-script                                                          x86_64                                                          26.1.19-1.2.0.romtest.1.xcpng8.3                                                              xcp-ng-ysionneau1                                                          2.5 M
 xapi-tests                                                                   x86_64                                                          26.1.19-1.2.0.romtest.1.xcpng8.3                                                              xcp-ng-ysionneau1                                                          5.0 M
 xapi-xe                                                                      x86_64                                                          26.1.19-1.2.0.romtest.1.xcpng8.3                                                              xcp-ng-ysionneau1                                                          1.5 M
 xcp-networkd                                                                 x86_64                                                          26.1.19-1.2.0.romtest.1.xcpng8.3                                                              xcp-ng-ysionneau1                                                          4.9 M
 xcp-rrdd                                                                     x86_64                                                          26.1.19-1.2.0.romtest.1.xcpng8.3                                                              xcp-ng-ysionneau1                                                          3.6 M
 xenopsd                                                                      x86_64                                                          26.1.19-1.2.0.romtest.1.xcpng8.3                                                              xcp-ng-ysionneau1                                                          1.4 M
 xenopsd-cli                                                                  x86_64                                                          26.1.19-1.2.0.romtest.1.xcpng8.3                                                              xcp-ng-ysionneau1                                                          2.0 M
 xenopsd-xc                                                                   x86_64                                                          26.1.19-1.2.0.romtest.1.xcpng8.3                                                              xcp-ng-ysionneau1                                                          5.3 M

Transaction Summary
=============================================================================================================================================================================================================================================================================================================================
Upgrade  22 Packages

• Then you will have to manually dump your iGPU VBIOS to precisely this path: /root/vbios.bin
  • for AMD modern iGPU it should be located in an ACPI table called VFCT, you could first call acpidump -b then use this """one liner"""

python3 -c 'import struct,sys;d=open(sys.argv[1],"rb").read();assert d[:4]==b"VFCT",d[:4];n=struct.unpack("<I",d[4:8])[0];off=struct.unpack("<I",d[0x34:0x38])[0];i=0
while off and off+28<=n:
  pcibus,pcidev,pcifunc,vendor,device,ssvid,ssid,rev,imagelen=struct.unpack("<IIIHHHHII",d[off:off+28])
  if imagelen==0 or off+28+imagelen>n:break
  fn="vbios_%02d_%04x_%04x.bin"%(i,vendor,device);open(fn,"wb").write(d[off+28:off+28+imagelen]);print(fn,"bus=%d dev=%d func=%d vid=%04x did=%04x len=%d"%(pcibus,pcidev,pcifunc,vendor,device,imagelen));off+=28+imagelen;i+=1' vfct.dat

• Reboot the host
• Follow the usual PCI Passthrough procedure to pass-through the iGPU PCI device to a Linux VM (a Debian 13 for instance): https://docs.xcp-ng.org/compute/#pci-passthrough
• Make sure you have the corresponding gpu firmware Debian package installed in your VM: firmware-amd-graphics or firmware-intel-graphics
• Start the VM
• Check in the dmesg if the GPU is correctly detected and probed by the driver (amdgpu for AMD for instance)

What to report
Report as replies to this thread, putting text files as attached-files

• If it worked or not
• the dmesg | grep -E 'drm|amdgpu' of the Linux VM
• the xl dmesg of the host (dom0)
• the output of grep xen-pci-passthrough /var/log/xensource.log*
• the lspci -n -vvv -s SBDF (replace SBDF with the domain:bus:device.function) or your GPU, both in the host (dom0) and in the VM (beware the SBDF will be different!)
• hexdump -C /root/vbios.bin | head -n100 on the host (dom0)

Thank you for your help!

I get out of my silence to confirm that I'm indeed working on AMD MxGPU driver and AMD iGPU passthrough issues.
I confess that I have not communicated since a while on this work, but I'm making progress and I'm very close to publish a call to testing PoC packages that should improve the iGPU passthrough situation.

@steff22
I am making good progress on this and I will soon have some packages to share for people to try and give feedback

Just FYI, we had some issues with some Debian VMs for instance, with old kernel
where the VM would suddenly take 100% CPU and would we completely frozen and unresponsive.
It was in fact Debian enabling the suspend, the guest would suspend and trigger a bug in xen PV driver in the guest kernel which would never be able to wake up anymore.
Newer kernel don't have this issue and are able to wake up.
I'm quoting Debian here but it might very well happen on other distro

@PessimistTech Let's try to see if we have errors in the logs.

Run ls /boot/xen* and note the name of the xen file that ends with -d.gz

Then, can you reboot and , while in the grub menu, edit the XCP-ng menu entry (the first one) and make the following modifications:

• modify the Xen ELF loaded by multiboot2 to load the debug Xen build by specifying the file that ends with -d.gz instead of /boot/xen.gz
• on the same line, at the end, add iommu=debug

Then boot (by doing ctrl+X) and once booted and the VM with PCI passthrough started, please issue xl dmesg command in dom0 shell and report the content here.

Also please paste the result of dom0 dmesg.

then also an lspci -vvv -nn -s SBDF from both dom0 and the VM

replace SBDF with the segment bus device function address of the device (beware it won't be the same in dom0 and inside the VM).

Regards,

Yann

Maybe let's ping the entire @Team-Hypervisor-Kernel here

Hi @pessimisttech let me ping @Team-Hypervisor-Kernel here

Hello @mgr42
I can confirm that indeed today AMD iGPU passthrough is not working on XCP-ng.
I am currently (slowly) working on it.

There are other references about this topic in the Forum and on GitHub FYI:

• https://xcp-ng.org/forum/topic/11918/amd-barcelo-passthrough-issues-any-success-stories
• https://xcp-ng.org/forum/topic/8909/issue-to-load-gpu-passthrough-invalid-pci-rom-header-signature-expecting-0xaa55-got-0x4556
• https://github.com/xcp-ng/xcp/issues/806
• https://github.com/xapi-project/xen-api/commit/930ffb34e42d674b8234c3d8399fdba97efa6b38

On the setup I am working on, the Expansion ROM is provided indeed by firmware via the VFCT ACPI table.
I dumped it to a file and provided it to qemu-dm xen-pci-passthrough device via the romfile= argument (via a modified XAPI build).

I can now indeed see a ROM bar in the guest but unfortunately its content is not the content of the file therefore nothing works (amdgpu driver is unable to get its ROM and fails).

So now I'm debugging qemu to try to understand the issue with the emulated/passed-through ROM BAR.

Regards,

Yann
Kernel & Hypervisor team